检索结果-内蒙古大学图书馆

Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Congzhi Zhang, Linhai Wu, Jialong He, Yulan Zhou, Deyu School of Computer Science and Engineering Key Laboratory of Computer Network and Information Integration Ministry of Education Southeast University China Department of Informatics King’s College London United Kingdom The Alan Turing Institute United Kingdom

Despite the notable advancements of existing prompting methods, such as In-Context Learning and Chain-of-Thought for Large Language Models (LLMs), they still face challenges related to various biases. Traditional debiasing methods primarily focus on the model training stage, including approaches based on data augmentation and reweighting, yet they struggle with the complex biases inherent in LLMs. To address such limitations, the causal relationship behind the prompting methods is uncovered using a structural causal model, and a novel causal prompting method based on front-door adjustment is proposed to effectively mitigate LLMs biases. In specific, causal intervention is achieved by designing the prompts without accessing the parameters and logits of LLMs. The chain-of-thought generated by LLM is employed as the mediator variable and the causal effect between input prompts and output answers is calculated through front-door adjustment to mitigate model biases. Moreover, to accurately represent the chain-of-thoughts and estimate the causal effects, contrastive learning is used to fine-tune the encoder of chain-of-thought by aligning its space with that of the LLM. Experimental results show that the proposed causal prompting approach achieves excellent performance across seven natural language processing datasets on both open-source and closed-source LLMs. Copyright © 2024, The Authors. All rights reserved.

关键词： Contrastive Learning

SCOPE: Optimizing key-Value Cache Compression in Long-context Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Jialong Wang, Zhenglin Zhang, Linhai Lai, Yilong He, Yulan Zhou, Deyu School of Computer Science and Engineering Key Laboratory of Computer Network and Information Integration Ministry of Education Southeast University China Department of Informatics King’s College London United Kingdom The Alan Turing Institute United Kingdom

key-Value (KV) cache has become a bottleneck of LLMs for long-context generation. Despite the numerous efforts in this area, the optimization for the decoding phase is generally ignored. However, we believe such optimization is crucial, especially for long-output generation tasks based on the following two observations: (i) Excessive compression during the prefill phase which requires specific full context, impairs the comprehension of the reasoning task;(ii) Deviation of heavy hitters1 occurs in the reasoning tasks with long outputs. Therefore, SCOPE, a simple yet efficient framework that separately performs KV cache optimization during the prefill and decoding phases, is introduced. Specifically, the KV cache during the prefill phase is preserved to maintain the essential information, while a novel strategy based on sliding is proposed to select essential heavy hitters for the decoding phase. Memory usage and memory transfer are further optimized using adaptive and discontinuous strategies. Extensive experiments on LONGGENBENCH show the effectiveness and generalization of SCOPE and its compatibility as a plug-in to other prefill-only KV compression methods. 2 Copyright © 2024, The Authors. All rights reserved.

关键词： Decoding

Dynamic Heterogeneous Federated Learning with Multi-Level Prototypes

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Guo, Shunxin Wang, Hongsong Geng, Xin School of Computer Science and Engineering Southeast University Nanjing210096 China MOE Key Laboratory of Computer Network and Information Integration China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications Southeast University Ministry of Education China

Federated learning shows promise as a privacy-preserving collaborative learning technique. Existing research mainly focuses on skewing the label distribution across clients. However, most approaches suffer from catastrophic forgetting and classifier shift, mainly when the global distribution of all classes is extremely unbalanced and the data distribution of the client dynamically evolves over time. In this paper, we study the Dynamic Heterogeneous Federated Learning, which addresses the practical scenario where heterogeneous data distributions exist among different clients and dynamic tasks within the client. Accordingly, we propose a novel federated learning framework named Federated Multi-Level Prototypes and design federated multi-level regularizations. To mitigate classifier shift, we construct semantic prototypes to provide fruitful generalization knowledge. To maintain the model stability and consistency convergence, three regularizations are introduced as training losses, i.e., prototype-based regularization, semantic prototype-based regularization, and federated inter-task regularization. Extensive experiments show that the proposed method achieves advanced performance in various settings. © 2024, The Authors. All rights reserved.

关键词： Semantics

Disambiguated attention embedding for multi-instance partial-label learning 23

学校读者我要写书评

暂无评论

Disambiguated attention embedding for multi-instance partial...

Proceedings of the 37th International Conference on Neural information Processing Systems

作者： Wei Tang Weijia Zhang Min-Ling Zhang School of Computer Science and Engineering Southeast University Nanjing China and Key Laboratory of Computer Network and Information Integration (Southeast University) Ministry of Education China School of Information and Physical Sciences The University of Newcastle Callaghan NSW Australia

In many real-world tasks, the concerned objects can be represented as a multi-instance bag associated with a candidate label set, which consists of one ground-truth label and several false positive labels. Multi-instance partial-label learning (MIPL) is a learning paradigm to deal with such tasks and has achieved favorable performances. Existing MIPL approach follows the instance-space paradigm by assigning augmented candidate label sets of bags to each instance and aggregating bag-level labels from instance-level labels. However, this scheme may be suboptimal as global bag-level information is ignored and the predicted labels of bags are sensitive to predictions of negative instances. In this paper, we study an alternative scheme where a multi-instance bag is embedded into a single vector representation. Accordingly, an intuitive algorithm named DEMIPL, i.e., Disambiguated attention Embedding for Multi-Instance Partial-Label learning, is proposed. DEMIPL employs a disambiguation attention mechanism to aggregate a multi-instance bag into a single vector representation, followed by a momentum-based disambiguation strategy to identify the ground-truth label from the candidate label set. Furthermore, we introduce a real-world MIPL dataset for colorectal cancer classification. Experimental results on benchmark and real-world datasets validate the superiority of DEMIPL against the compared MIPL and partial-label learning approaches.

关键词：

Task allocation based on profit maximization for mobile crowdsourcing

学校读者我要写书评

暂无评论

The Journal of China Universities of Posts and Telecommunications 2020年第1期27卷 26-37页

作者： Hou Yinglin Cheng Weiqing School of Computer Science Nanjing University of Posts and TelecommunicationsNanjing 210023China Key Laboratory of Computer Network and Information Integration Southeast UniversityNanjing 211189China

In recent years, with the development of smart devices, mobile users can use them to sense the environment. In order to improve the data quality and achieve maximum profits, incentive mechanism is needed to motivate users to participate. In this paper, reputation mechanism, participant selection, task allocation and joint pricing in mobile crowdsourcing system are studied. A user reputation evaluation method is proposed, and a participant selection algorithm(PSA) based on user reputation is proposed. Besides, a social welfare maximization algorithm(SWMA) is proposed, which achieves task pricing with maximizing the interests of all parties, including both task publishers and mobile users. The social welfare maximization problem is divided into local optimization sub-problems which can be solved by double decomposition. It is proved that the algorithm converges to the optimal solution. Results of simulations verify that algorithms PSA and SWMA are effective.

关键词： mobile crowdsourcing maximum profits reputation mechanism task allocation

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Shi, Jiang-Xin Zhang, Chi Wei, Tong Li, Yu-Feng National Key Laboratory for Novel Software Technology School of Artificial Intelligence Nanjing University China National Key Laboratory for Novel Software Technology Nanjing University China School of Computer Science and Engineering Key Laboratory of Computer Network and Information Integration of Ministry of Education Southeast University China

Pre-trained vision-language models like CLIP have shown powerful zero-shot inference ability via image-text matching and prove to be strong few-shot learners in various downstream tasks. However, in real-world scenarios, adapting CLIP to downstream tasks may encounter the following challenges: 1) data may exhibit long-tailed data distributions and might not have abundant samples for all the classes;2) There might be emerging tasks with new classes that contain no samples at all. To overcome them, we propose a novel framework to achieve efficient and long-tailed generalization, which can be termed as Candle. During the training process, we propose compensating logit-adjusted loss to encourage large margins of prototypes and alleviate imbalance both within the base classes and between the base and new classes. For efficient adaptation, we treat the CLIP model as a black box and leverage the extracted features to obtain visual and textual prototypes for prediction. To make full use of multi-modal information, we also propose cross-modal attention to enrich the features from both modalities. For effective generalization, we introduce virtual prototypes for new classes to make up for their lack of training images. Candle achieves state-of-the-art performance over extensive experiments on 11 diverse datasets while substantially reducing the training time, demonstrating the superiority of our approach. The source code is available at https://***/shijxcs/Candle. Copyright © 2024, The Authors. All rights reserved.

关键词： Zero-shot learning

In-band network-Wide Telemetry for Topology-Varying LEO Satellite networks

学校读者我要写书评

暂无评论

In-band Network-Wide Telemetry for Topology-Varying LEO Sate...

IEEE Conference on Global Communications (GLOBECOM)

作者： Yan Zhang Tian Pan Qiang Fu Yan Zheng Xinyu Feng Jiang Liu Haipeng Yao Tao Huang Purple Mountain Laboratories Nanjing China Beijing University of Posts and Telecommunications Beijing China Ministry of Education Key Laboratory of Computer Network and Information Integration (Southeast University) China RMIT University Melbourne Australia

ISBN: (数字)9798350351255

ISBN: (纸本)9798350351262

Driven by technological advances and new business models, we have seen a renewed interest in LEO satellite constellations. The deployment of large-scale LEO satellite networks is becoming a reality. The network topology changes periodically, as satellites orbit the Earth. This imposes a great challenge to network monitoring. Meanwhile, as a new network monitoring method, In-band network Telemetry (INT) can provide per-hop granular telemetry metadata, needed to tackle the mobile nature of LEO satellite constellations. Given this, we apply INT to LEO satellite networks for real-time fine-grained monitoring. We propose a path planning solution to identify the paths for network-wide telemetry and the paths for disseminating the telemetry data to the ground facilities. By taking advantage of the predictable satellite trajectories and topology variations, the path planning solution is designed to achieve network-wide coverage and minimize telemetry overhead. We take the LEO48 constellation as an example to visually show the detailed paths of the monitoring scheme. We conduct experiments on different sizes of networks to evaluate the original path planning algorithm and the improved balanced algorithm in this paper, demonstrating the timeliness and balance of the telemetry solution.

关键词： Satellite constellations Visualization Satellites network topology Low earth orbit satellites Prediction algorithms Real-time systems Trajectory Telemetry Monitoring

Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Zi-Hao Fang, Siyuan Zhou, Zi-Jing Wei, Tong Wan, Yuanyu Zhang, Min-Ling School of Computer Science and Engineering Southeast University Nanjing China Key Laboratory of Computer Network and Information Integration Southeast University Ministry of Education China Xiaomi Inc. China School of Software Technology Zhejiang University Ningbo China

Long-tailed semi-supervised learning poses a significant challenge in training models with limited labeled data exhibiting a long-tailed label distribution. Current state-of-the-art LTSSL approaches heavily rely on high-quality pseudo-labels for large-scale unlabeled data. However, these methods often neglect the impact of representations learned by the neural network and struggle with real-world unlabeled data, which typically follows a different distribution than labeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. Our framework derives the class-balanced contrastive loss through Gaussian kernel density estimation. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using reliable and smoothed pseudo-labels. By progressively estimating the underlying label distribution and optimizing its alignment with model predictions, we tackle the diverse distribution of unlabeled data in real-world scenarios. Extensive experiments across multiple datasets with varying unlabeled data distributions demonstrate that CCL consistently outperforms prior state-of-the-art methods, achieving over 4% improvement on the ImageNet-127 dataset. Our source code is available at https://***/zhouzihao11/CCL. © 2024, CC BY.

关键词： Contrastive Learning

Vision-Language Models are Strong Noisy Label Detectors

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wei, Tong Li, Hao-Tian Li, Chun-Shu Shi, Jiang-Xin Li, Yu-Feng Zhang, Min-Ling School of Computer Science and Engineering Southeast University Nanjing China Key Laboratory of Computer Network and Information Integration Southeast University Ministry of Education China National Key Laboratory for Novel Software Technology Nanjing University China School of Artificial Intelligence Nanjing University China

Recent research on fine-tuning vision-language models has demonstrated impressive performance in various downstream tasks. However, the challenge of obtaining accurately labeled data in real-world applications poses a significant obstacle during the fine-tuning process. To address this challenge, this paper presents a Denoising Fine-Tuning framework, called DEFT, for adapting vision-language models. DEFT utilizes the robust alignment of textual and visual features pre-trained on millions of auxiliary image-text pairs to sieve out noisy labels. The proposed framework establishes a noisy label detector by learning positive and negative textual prompts for each class. The positive prompt seeks to reveal distinctive features of the class, while the negative prompt serves as a learnable threshold for separating clean and noisy samples. We employ parameter-efficient fine-tuning for the adaptation of a pre-trained visual encoder to promote its alignment with the learned textual prompts. As a general framework, DEFT can seamlessly fine-tune many pre-trained models to downstream tasks by utilizing carefully selected clean samples. Experimental results on seven synthetic and real-world noisy datasets validate the effectiveness of DEFT in both noisy label detection and image classification tasks. Our source code is available at https://***/HotanLee/DeFT. © 2024, CC BY.

关键词： Visual languages