检索结果-内蒙古大学图书馆

Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Li, Kewei Kong, Yanwen Xu, Yiping Su, Jianlin Huang, Lan Zhang, Ruochi Zhou, Fengfeng College of Computer Science and Technology Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University Jilin Changchun130012 China Moonshot AI Ltd. Beijing100086 China

Since the emergence of research on improving the length extrapolation capabilities of large models in 2021, some studies have made modifications to the scaling factor in the scaled dot-product attention mechanism as part of their proposed methods without rigorous theoretical justifications. To fill this gap, we propose two new scaled temperatures based on entropy invariance to enhance length extrapolation. First, a training-free method InfoScale is designed for dot-product attention, and preserves focus on original tokens during length extrapolation by ensuring consistent entropy. Second, we theoretically analyze the impact of scaling (CosScale) on cosine attention. Experimental data demonstrates that combining InfoScale and CosScale achieves state-of-the-art performance on the GAU-α model with a context window extended to 64 times the training length, and outperforms seven existing methods. Our analysis reveals that significantly increasing CosScale approximates the Windowed Attention, and highlights the significance of attention score dilution as a key challenge in long-range context handling. The code and data are available at https://***/HT-NEKO/InfoScale. © 2025, CC BY-NC-ND.

关键词： Data handling

Instance-adaptive zero-shot chain-of-thought prompting 24

学校读者我要写书评

暂无评论

Instance-adaptive zero-shot chain-of-thought prompting

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Xiaosong Yuan Chen Shen Shaotian Yan Xiaofeng Zhang Liang Xie Wenxiao Wang Renchu Guan Ying Wang Jieping Ye College of Computer Science and Technology Jilin University and Key Laboratory of Symbolic Computation and Knowledge Engineering MoE Jilin University Alibaba Cloud Computing Shanghai Jiao Tong University College of Computer Science and Technology Zhejiang University of Technology College of Software Zhejiang University

ISBN: (纸本)9798331314385

Zero-shot Chain-of-Thought (CoT) prompting emerges as a simple and effective strategy for enhancing the performance of large language models (LLMs) in real-world reasoning tasks. Nonetheless, the efficacy of a singular, task-level prompt uniformly applied across the whole of instances is inherently limited since one prompt cannot be a good partner for all, a more appropriate approach should consider the interaction between the prompt and each instance meticulously. This work introduces an instance-adaptive prompting algorithm as an alternative zero-shot CoT reasoning scheme by adaptively differentiating good and bad prompts. Concretely, we first employ analysis on LLMs through the lens of information flow to detect the mechanism under zero-shot CoT reasoning, in which we discover that information flows from question to prompt and question to rationale jointly influence the reasoning results most. We notice that a better zero-shot CoT reasoning needs the prompt to obtain semantic information from the question, and then the rationale aggregates sufficient information from the question directly and via the prompt indirectly. On the contrary, lacking any of those would probably lead to a bad one. Stem from that, we further propose an instance-adaptive prompting strategy (IAP) for zero-shot CoT reasoning. Experiments conducted with LLaMA-2, LLaMA-3, and Qwen on math, logic, and commonsense reasoning tasks (e.g., GSM8k, MMLU, Causal Judgement) obtain consistent improvement, demonstrating that the instance-adaptive zero-shot CoT prompting performs better than other task-level methods with some curated prompts or sophisticated procedures, showing the significance of our findings in the zero-shot CoT reasoning mechanism.

关键词：

UAV-assisted Joint Mobile Edge computing and Data Collection via Matching-enabled Deep Reinforcement Learning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wang, Boxiong Kang, Hui Li, Jiahui Sun, Geng Sun, Zemin Wang, Jiacheng Niyato, Dusit College of Computer Science and Technology Jilin University Changchun130012 China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University Changchun130012 China College of Computing and Data Science Nanyang Technological University 639798 Singapore College of Computing and Data Science Nanyang Technological University Singapore

Unmanned aerial vehicle (UAV)-assisted mobile edge computing (MEC) and data collection (DC) have been popular research issues. Different from existing works that consider MEC and DC scenarios separately, this paper investigates a multi-UAV-assisted joint MEC-DC system. Specifically, we formulate a joint optimization problem to minimize the MEC latency and maximize the collected data volume. This problem can be classified as a non-convex mixed integer programming problem that exhibits long-term optimization and dynamics. Thus, we propose a deep reinforcement learning-based approach that jointly optimizes the UAV movement, user transmit power, and user association in real time to solve the problem efficiently. Specifically, we reformulate the optimization problem into an action space-reduced Markov decision process (MDP) and optimize the user association by using a two-phase matching-based association (TMA) strategy. Subsequently, we propose a soft actor-critic (SAC)-based approach that integrates the proposed TMA strategy (SAC-TMA) to solve the formulated joint optimization problem collaboratively. Simulation results demonstrate that the proposed SAC-TMA is able to coordinate the two subsystems and can effectively reduce the system latency and improve the data collection volume compared with other benchmark algorithms. Copyright © 2025, The Authors. All rights reserved.

关键词： Unmanned aerial vehicles (UAV)

A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Xiaoli Wang, Liying Zhao, Libo Li, Xiongfei Ma, Siwei The Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University Changchun130012 China The Institute of Digital Media School of Electronic Engineering and Computer Science Peking University Beijing100871 China

Multi-modality image fusion aims at fusing modality-specific (complementarity) and modality-shared (correlation) information from multiple source images. To tackle the problem of the neglect of inter-modal information, high-frequency information loss and limited attention to downstream tasks, this paper focuses on how to model correlation-driven decomposing features and reason high-level graph representation by efficiently extracting complementary features and aggregating multi-guided features. We propose a three-branch encoder-decoder architecture along with corresponding fusion layers as the fusion strategy. The transformer with Multi-Dconv Head Transposed Attention and Local-enhanced Feed Forward network is used to extract shallow features after the depthwise convolution. In the three parallel branches of the encoder, Cross Attention and Invertible Block (CAI) enables to extract local features and preserve high-frequency texture details. Base Feature Extraction Module (BFE) with residual connections can capture long-range dependencies and enhance modality-shared information. Graph Reasoning Module (GR) is introduced to reason high-level cross-modality relations and simultaneously extract low-level detail features as CAI’s modality-specific complementary information. Experiments demonstrate that our method obtains competitive results compared with state-of-the-art methods in visible/infrared image fusion and medical image fusion tasks. Moreover, we surpass other fusion methods in terms of subsequent tasks, averagely scoring 8.27% mAP@0.5 higher in object detection and 5.85% mIoU higher in semantic segmentation. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantic Segmentation

DC-Instruct: An Effective Framework for Generative Multi-intent Spoken Language Understanding

学校读者我要写书评

暂无评论

DC-Instruct: An Effective Framework for Generative Multi-int...

2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024

作者： Xing, Bowen Liao, Lizi Huang, Minlie Tsang, Ivor W. Beijing Key Laboratory of Knowledge Engineering for Materials Science School of Computer and Communication Engineering University of Science and Technology Beijing China Singapore Management University Singapore Group Tsinghua University China CFAR and IHPC Agency for Science Technology and Research Singapore College of Computing and Data Science Nanyang Technological University Singapore

ISBN: (纸本)9798891761643

In the realm of multi-intent spoken language understanding, recent advancements have leveraged the potential of prompt learning frameworks. However, critical gaps exist in these frameworks: the lack of explicit modeling of dual-task dependencies and the oversight of task-specific semantic differences among utterances. To address these shortcomings, we propose DC-Instruct, a novel generative framework based on Dual-task Inter-dependent Instructions (DII) and Supervised Contrastive Instructions (SCI). Specifically, DII guides large language models (LLMs) to generate labels for one task based on the other task's labels, thereby explicitly capturing dual-task inter-dependencies. Moreover, SCI leverages utterance semantics differences by guiding LLMs to determine whether a pair of utterances share the same or similar labels. This can improve LLMs on extracting and discriminating task-specific semantics, thus enhancing their SLU reasoning abilities. Extensive experiments on public benchmark datasets show that DC-Instruct markedly outperforms current generative models and state-of-the-art methods, demonstrating its effectiveness in enhancing dialogue language understanding and reasoning. © 2024 Association for Computational Linguistics.

关键词： Computational linguistics

Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables

学校读者我要写书评

暂无评论

Learning Semantic Textual Similarity via Topic-informed Disc...

2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

作者： Yu, Erxin Du, Lan Jin, Yuan Wei, Zhepei Chang, Yi School of Artificial Intelligence Jilin University China Key Laboratory of Symbolic Computation and Knowledge Engineering Jilin University China International Center of Future Science Jilin University China Faculty of Information Technology Monash University Australia

Recently, discrete latent variable models have received a surge of interest in both Natural Language Processing (NLP) and Computer Vision (CV), attributed to their comparable performance to the continuous counterparts in representation learning, while being more interpretable in their predictions. In this paper, we develop a topic-informed discrete latent variable model for semantic textual similarity, which learns a shared latent space for sentence-pair representation via vector quantization. Compared with previous models limited to local semantic contexts, our model can explore richer semantic information via topic modeling. We further boost the performance of semantic similarity by injecting the quantized representation into a transformer-based language model with a well-designed semantic-driven attention mechanism. We demonstrate, through extensive experiments across various English language datasets, that our model is able to surpass several strong neural baselines in semantic textual similarity tasks. © 2022 Association for Computational Linguistics.

关键词： Semantics

In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Huang, Sili Hu, Jifeng Chen, Hechang Sun, Lichao Yang, Bo School of Artificial Intelligence Jilin University China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University China Lehigh University BethlehemPA United States

In-context learning is a promising approach for offline reinforcement learning (RL) to handle online tasks, which can be achieved by providing task prompts. Recent works demonstrated that in-context RL could emerge with self-improvement in a trial-and-error manner when treating RL tasks as an across-episodic sequential prediction problem. Despite the self-improvement not requiring gradient updates, current works still suffer from high computational costs when the across-episodic sequence increases with task horizons. To this end, we propose an In-context Decision Transformer (IDT) to achieve self-improvement in a high-level trial-and-error manner. Specifically, IDT is inspired by the efficient hierarchical structure of human decision-making and thus reconstructs the sequence to consist of high-level decisions instead of low-level actions that interact with environments. As one high-level decision can guide multi-step low-level actions, IDT naturally avoids excessively long sequences and solves online tasks more efficiently. Experimental results show that IDT achieves state-of-the-art in long-horizon tasks over current in-context RL methods. In particular, the online evaluation time of our IDT is 36× times faster than baselines in the D4RL benchmark and 27× times faster in the Grid World benchmark. Copyright © 2024, The Authors. All rights reserved.

关键词： Reinforcement learning

Affine Invariant Feature Based on Local Color

学校读者我要写书评

暂无评论

Chinese Journal of Electronics 2023年第3期22卷 511-516页

作者： LIU Pingping ZHOU Qiuzhan ZHAO Hongwci School of Computer Science and Technology Jilin University Changchun China Ministry of Education Key Laboratory of Symbolic Computation and Knowledge Engineering Jilin University Changchun China School of Communication Engineering Jilin University Changchun China

Image localization using local features has attracted a lot of attention in recent mobile robots research. A novel, fast local invariant feature in affine transformation is proposed in this article, called AIFLC (Affine invariant feature based on local color). We adopt affine moment invariants to build affine invariant descriptors. Moreover, we use color gradient based center moment instead of original pixel values in order to enhance discriminative power and robustness of descriptor in photometric transformations. Simulation results show that the run time of AIFLC using optimal selection parameters is about 1/3 of classical SIFT algorithm. Using the standard evaluation images and the ones taken by mobile robots, we experimentally demonstrate that the AIFLC outperforms the state-of-art approaches such as SIFT and SURF in terms of image scaling, rotation, viewpoint changing, and blur transformations.

关键词： Location awareness Image color analysis Navigation Simulation Lighting Real-time systems Robustness

Few-Shot Data Completion for New Tasks in Sparse Crowdsensing

学校读者我要写书评

暂无评论

Few-Shot Data Completion for New Tasks in Sparse Crowdsensin...

IEEE Annual Joint Conference: INFOCOM, IEEE Computer and Communications Societies

作者： En Wang Mijia Zhang Bo Yang Yang Xu Zixuan Song Yongjian Yang College of Computer Science and Technology Jilin University China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University China College of Computer Science and Electronic Engineering Hunan University China

ISBN: (数字)9798350383508

ISBN: (纸本)9798350383515

Mobile Crowdsensing is a type of technology that utilizes mobile devices and volunteers to gather data about specific topics at large scales in real-time. However, in practice, limited participation leads to missing data, i.e., the collected data may be sparse, which makes it difficult to perform accurate analysis. A possible technique called sparse crowdsensing incorporates the sparse case with data completion, where unsensed data could be estimated through inference. However, sparse crowdsensing typically suffers from poor performance during the data completion stage due to various challenges: the sparsity of the sensed data, reliance on numerous timeslots, and uncertain spatiotemporal connections. To resolve such few-shot issues, the proposed solution uses the Correlated Data Fusion for Matrix Completion (CDFMC) approach, which leverages a small amount of objective data to retrain an auxiliary dataset-based pre-trained model that can estimate unsensed data efficiently. CDFMC is trained using a combination of the traditional Deep Matrix Factorization and the Kalman Filtering, which not only enables the efficient representation and comparison of data samples but also fuses the objective data and auxiliary data effectively. Evaluation results show that the proposed CDFMC outperforms baseline techniques, achieving high accuracy in completing unsensed data with minimal training data.

关键词： Performance evaluation Accuracy Crowdsensing Filtering Training data Data models Real-time systems