检索结果-内蒙古大学图书馆

BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Shaolei Zhang, Kehao Fang, Qingkai Guo, Shoutao Zhou, Yan Liu, Xiaodong Feng, Yang Key Laboratory of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences ICT CAS China Key Laboratory of AI Safety Chinese Academy of Sciences China University of Chinese Academy of Sciences Beijing China Research Center of Distributed Systems Institute of Computing Technology Chinese Academy of Sciences ICT CAS China

Large language models (LLMs), with their powerful generative capabilities and vast knowledge, empower various tasks in everyday life. However, these abilities are primarily concentrated in high-resource languages, leaving low-resource languages with weaker generative capabilities and relatively limited knowledge. Enhancing the multilingual capabilities of LLMs is therefore crucial for serving over 100 linguistic communities worldwide. An intuitive approach to enhance the multilingual capabilities would be to construct instruction data for various languages, but constructing instruction data for over 100 languages is prohibitively costly. In this paper, we introduce BayLing 2, which efficiently transfers generative capabilities and knowledge from high-resource languages to low-resource languages through language alignment. To achieve this, we constructed a dataset of 3.2 million instructions, comprising high-resource language instructions (Chinese and English) and cross-lingual instructions for 100+ languages and performed instruction tuning based on the dataset to facilitate the capability transfer between languages. Using Llama as the foundation model, we developed BayLing-2-7B, BayLing-2-13B, and BayLing-2-8B, and conducted a comprehensive evaluation of BayLing. For multilingual translation across 100+ languages, BayLing shows superior performance compared to open-source models of similar scale. For multilingual knowledge and understanding benchmarks, BayLing achieves significant improvements across over 20 low-resource languages, demonstrating its capability of effective knowledge transfer from high-resource to low-resource languages. Furthermore, results on English benchmarks indicate that BayLing maintains high performance in high-resource languages while enhancing the performance in low-resource languages. Demo2, homepage3, code4 and models5of BayLing are available. © 2024, CC BY-NC-ND.

关键词： Benchmarking

Revisiting Robust RAG: Do We Still Need Complex Robust Training in the Era of Powerful LLMs?

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Ding, Hanxing Tao, Shuchang Pang, Liang Wei, Zihao Chen, Liwei Xu, Kun Shen, Huawei Cheng, Xueqi Key Laboratory of AI Safety Institute of Computing Technology Chinese Academy of Sciences China University of Chinese Academy of Sciences China Kuaishou Technology China

Retrieval-augmented generation (RAG) systems often suffer from performance degradation when encountering noisy or irrelevant documents, driving researchers to develop sophisticated training strategies to enhance their robustness against such retrieval noise. However, as large language models (LLMs) continue to advance, the necessity of these complex training methods is increasingly questioned. In this paper, we systematically investigate whether complex robust training strategies remain necessary as model capacity grows. Through comprehensive experiments spanning multiple model architectures and parameter scales, we evaluate various document selection methods and adversarial training techniques across diverse datasets. Our extensive experiments consistently demonstrate that as models become more powerful, the performance gains brought by complex robust training methods drop off dramatically. We delve into the rationale and find that more powerful models inherently exhibit superior confidence calibration, better generalization across datasets (even when trained with randomly selected documents), and optimal attention mechanisms learned with simpler strategies. Our findings suggest that RAG systems can benefit from simpler architectures and training strategies as models become more powerful, enabling more scalable applications with minimal complexity. Copyright © 2025, The Authors. All rights reserved.

关键词：

Adaptive Token Biaser: Knowledge Editing via Biasing key Entities

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Bi, Baolong Liu, Shenghua Wang, Yiwei Mei, Lingrui Gao, Hongcheng Xu, Yilong Cheng, Xueqi CAS Key Laboratory of AI Safety Institute of Computing Technology Chinese Academy of Sciences China University of Chinese Academy of Sciences China University of California Los Angeles United States

The parametric knowledge memorized by large language models (LLMs) becomes outdated quickly. In-context editing (ICE) is currently the most effective method for updating the knowledge of LLMs. Recent advancements involve enhancing ICE by modifying the decoding strategy, obviating the need for altering internal model structures or adjusting external prompts. However, this enhancement operates across the entire sequence generation, encompassing a plethora of non-critical tokens. In this work, we introduce Adaptive Token Biaser (ATBIAS), a new decoding technique designed to enhance ICE. It focuses on the tokens that are mostly related to knowledge during decoding, biasing their logits by matching key entities related to new and parametric knowledge. Experimental results show that ATBIAS significantly enhances ICE performance, achieving up to a 32.3% improvement over state-of-the-art ICE methods while incurring only half the latency. ATBIAS not only improves the knowledge editing capabilities of ICE but can also be widely applied to LLMs with negligible cost. Copyright © 2024, The Authors. All rights reserved.

关键词： Model structures

A THEORY FOR TOKEN-LEVEL HARMONIZATION IN RETRIEVAL-AUGMENTED GENERATION

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Xu, Shicheng Pang, Liang Shen, Huawei Cheng, Xueqi Key Laboratory of AI Safety Institute of Computing Technology Chinese Academy of Sciences China University of Chinese Academy of Sciences China

Retrieval-augmented generation (RAG) utilizes retrieved texts to enhance large language models (LLMs). Studies show that while RAG provides valuable external information (benefit), it may also mislead LLMs (detriment) with noisy or incorrect retrieved texts. Although many existing methods attempt to preserve benefit and avoid detriment, they lack a theoretical explanation for RAG. The benefit and detriment in the next token prediction of RAG remain a’black box’ that cannot be quantified or compared in an explainable manner, so existing methods are data-driven, need additional utility evaluators or post-hoc. This paper takes the first step towards providing a theory to explain and trade off the benefit and detriment in RAG. First, we model RAG as the fusion between distribution of LLM’s knowledge and distribution of retrieved texts. Then, we formalize the trade-off between the value of external knowledge (benefit) and its potential risk of misleading LLMs (detriment) in next token prediction of RAG by distribution difference in this fusion. Finally, we prove that the actual effect of RAG on the token, which is the comparison between benefit and detriment, can be predicted without any training or accessing the utility of retrieval. Based on our theory, we propose a practical novel method, Tok-RAG, which achieves collaborative generation between the pure LLM and RAG at token level to preserve benefit and avoid detriment. Experiments in real-world tasks using LLMs such as OPT, LLaMA-2, and Mistral show the effectiveness of our method and support our theoretical findings. Code is available1 Copyright © 2024, The Authors. All rights reserved.

关键词： Modeling languages

MITA: Bridging the Gap between Model and Data for Test-time Adaptation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yuan, Yige Xu, Bingbing Xiao, Teng Hou, Liang Sun, Fei Shen, Huawei Cheng, Xueqi CAS Key Laboratory of AI Safety Institute of Computing Technology Chinese Academy of Sciences China University of Chinese Academy of Sciences China Pennsylvania State University United States Kuaishou Technology China

Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. However, existing mainstream TTA methods, predominantly operating at batch level, often exhibit suboptimal performance in complex real-world scenarios, particularly when confronting outliers or mixed distributions. This phenomenon stems from a pronounced over-reliance on statistical patterns over the distinct characteristics of individual instances, resulting in a divergence between the distribution captured by the model and data characteristics. To address this challenge, we propose Meet-In-The-Middle based Test-Time Adaptation (MITA), which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions, thereby meeting in the middle. MITA pioneers a significant departure from traditional approaches that focus solely on aligning the model to the data, facilitating a more effective bridging of the gap between model’s distribution and data characteristics. Comprehensive experiments with MITA across three distinct scenarios (Outlier, Mixture, and Pure) demonstrate its superior performance over SOTA methods, highlighting its potential to significantly enhance generalizability in practical applications. Copyright © 2024, The Authors. All rights reserved.

关键词： Data assimilation

Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Deng, Jingcheng Jiang, Zhongtao Pang, Liang Chen, Liwei Xu, Kun Wei, Zihao Shen, Huawei Cheng, Xueqi Key Laboratory of AI Safety Institute of Computing Technology Chinese Academy of Sciences China University of Chinese Academy of Sciences China Kuaishou Technology China

A new trend uses LLMs as dense text encoders via contrastive learning. However, since LLM embeddings predict the probability distribution of the next token, they are inherently generative and distributive, conflicting with contrastive learning, which requires embeddings to capture full-text semantics and align via cosine similarity. This discrepancy hinders the full utilization of LLMs’ pre-training capabilities, resulting in inefficient learning. In response to this issue, we propose AutoRegEmbed, a new contrastive learning method built on embedding conditional probability distributions, which integrates two core tasks: information compression and conditional distribution alignment. The information compression task encodes text into the embedding space, ensuring that the embedding vectors capture global semantics. The conditional distribution alignment task focuses on aligning text embeddings with positive samples embeddings by leveraging the conditional distribution of embeddings while simultaneously reducing the likelihood of generating negative samples from text embeddings, thereby achieving embedding alignment and uniformity. Experimental results demonstrate that our method significantly outperforms traditional contrastive learning approaches and achieves performance comparable to state-of-the-art models when using the same amount of data. Our code is available at https://***/TrustedLLM/AutoRegEmbed Copyright © 2025, The Authors. All rights reserved.

关键词： Contrastive Learning

Source Echo Chamber: Exploring the Escalation of Source Bias in User, Data, and Recommender System Feedback Loop

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhou, Yuqi Dai, Sunhao Pang, Liang Wang, Gang Dong, Zhenhua Xu, Jun Wen, Ji-Rong Gaoling School of Artificial Intelligence Renmin University of China Beijing China CAS Key Laboratory of AI Safety Institute of Computing Technology CAS Beijing China Huawei Noah’s Ark Lab Shenzhen China

Recently, researchers have uncovered that neural retrieval models prefer ai-generated content (aiGC), called source bias [10, 40]. Compared to active search behavior, recommendation represents another important means of information acquisition, where users are more prone to source bias. Furthermore, delving into the recommendation scenario, as aiGC becomes integrated within the feedback loop involving users, data, and the recommender system, it progressively contaminates the candidate items, the user interaction history, and ultimately, the data used to train the recommendation models. How and to what extent the source bias affects the neural recommendation models within feedback loop remains unknown. In this study, we extend the investigation of source bias into the realm of recommender systems, specifically examining its impact across different phases of the feedback loop. We conceptualize the progression of aiGC integration into the recommendation content ecosystem in three distinct phases-HGC dominate, HGC-aiGC coexist, and aiGC dominance-each representing past, present, and future states, respectively. Through extensive experiments across three datasets from diverse domains, we demonstrate the prevalence of source bias and reveal a potential digital echo chamber with source bias amplification throughout the feedback loop. This trend risks creating a recommender ecosystem with limited information source, such as aiGC, being disproportionately recommended. To counteract this bias and prevent its escalation in the feedback loop, we introduce a black-box debiasing method that maintains model impartiality towards both HGC and aiGC. Our experimental results validate the effectiveness of the proposed debiasing method, confirming its potential to disrupt the feedback loop. © 2024, CC BY-NC-ND.

关键词： Recommender systems

学校读者我要写书评

暂无评论

Context Graph

arXiv 2024年

作者： Xu, Chengjin Li, Muzhi Yang, Cehao Jiang, Xuhui Tang, Lumingyuan Qi, Yiyan Guo, Jian IDEA Research International Digital Economy Academy China Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong CAS Key Laboratory of AI Safety Institute of Computing Technology CAS China

Knowledge Graphs (KGs) are foundational structures in many ai applications, representing entities and their interrelations through triples. However, triple-based KGs lack the contextual information of relational knowledge, like temporal dynamics and provenance details, which are crucial for comprehensive knowledge representation and effective reasoning. Instead, Context Graphs (CGs) expand upon the conventional structure by incorporating additional information such as time validity, geographic location, and source provenance. This integration provides a more nuanced and accurate understanding of knowledge, enabling KGs to offer richer insights and support more sophisticated reasoning processes. In this work, we first discuss the inherent limitations of triple-based KGs and introduce the concept of CGs, highlighting their advantages in knowledge representation and reasoning. We then present a context graph reasoning CGR3 paradigm that leverages large language models (LLMs) to retrieve candidate entities and related contexts, rank them based on the retrieved information, and reason whether sufficient information has been obtained to answer a query. Our experimental results demonstrate that CGR3 significantly improves performance on KG completion (KGC) and KG question answering (KGQA) tasks, validating the effectiveness of incorporating contextual information on KG representation and reasoning. © 2024, CC BY-NC-ND.

关键词： Knowledge graph

F2GNN: An Adaptive Filter with Feature Segmentation for Graph-Based Fraud Detection

学校读者我要写书评

暂无评论

F2GNN: An Adaptive Filter with Feature Segmentation for Grap...

International Conference on Acoustics, Speech, and Signal Processing (IcasSP)

作者： Guanghui Hu Yang Liu Qing He Xiang Ao Henan Institute of Advanced Technology Zhengzhou University Zhengzhou P.R. China Key Laboratory of AI Safety & Security Chinese Academy of Sciences (CAS) Institute of Computing Technology CAS Beijing China CASMINO Ltd. Suzhou China

Graph Neural Networks (GNNs) have received remarkable success in identifying fraudulent activities on graphs. Most approaches leverage the full user feature together and aggregate the messages from its neighbors by a graph filter. However, due to the adversarial activities like the camouflage of fraudsters, most dimensions of fraudsters' features resemble normal users, and modeling the features as a whole cannot fully explore the small-portion fraudulent features. In this paper, we attempt to segment the user features and apply adaptive graph filters on each segmentation for better modeling of fraudulent features. We propose an adaptive filter with feature segmentation (shortened as F 2 GNN) to alleviate these challenges. Experimental results on two real-world datasets demonstrate that F 2 GNN outperforms state-of-the-art baselines for graph-based fraud detection. In addition, the adaptive filter with feature segmentation can effectively address the class imbalance problem in the task of fraud detection.

关键词：