检索结果-内蒙古大学图书馆

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Patel, Divya Patel, Pathik Chander, Ankush Dasgupta, Sourish Chakraborty, Tanmoy KDM Lab Dhirubhai Ambani Institute of Information & Communication Technology India Indian Institute of Technology Delhi India

ISBN: (纸本)9798891761643

Large language Models (LLMs) have succeeded considerably in In-Context-Learning (ICL) based summarization. However, saliency is subject to the users' specific preference histories. Hence, we need reliable In-Context Personalization Learning (ICPL) capabilities within such LLMs. For any arbitrary LLM to exhibit ICPL, it needs to have the ability to discern contrast in user profiles. A recent study proposed a measure for degree-of-personalization called EGISES for the first time. EGISES measures a model's responsiveness to user profile differences. However, it cannot test if a model utilizes all three types of cues provided in ICPL prompts: (i) example summaries, (ii) user's reading histories, and (iii) contrast in user profiles. To address this, we propose the iCOPERNICUS framework, a novel In-Context Personalization Learning Scrutiny of Summarization capability in LLMs that uses EGISES as a comparative measure. As a case-study, we evaluate 17 state-of-the-art LLMs based on their reported ICL performances and observe that 15 models' ICPL degrades (min: 1.6%↓;max: 3.6%↓) when probed with richer prompts, thereby showing lack of true ICPL. © 2024 Association for Computational Linguistics.

关键词： Contrastive Learning

来源：评论

学校读者我要写书评

暂无评论

On Diversified Preferences of Large language Model Alignment

On Diversified Preferences of Large Language Model Alignment

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Zeng, Dun Dai, Yong Cheng, Pengyu Wang, Longyue Hu, Tianhao Chen, Wanshun Du, Nan Xu, Zenglin Tencent AI Lab China HiThink Research Singapore Alibaba Group China Peng Cheng Lab China

ISBN: (纸本)9798891761681

Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction ***, in this pluralistic world, human preferences can be diversified due to annotators' different tastes, which hinders the effectiveness of LLM alignment *** paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse *** analysis reveals that the impact of diversified human preferences depends on both model size and data *** models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate *** mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of ***, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared *** experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance. © 2024 Association for Computational Linguistics.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Evaluating the Instruction-Following Robustness of Large language Models to Prompt Injection

Evaluating the Instruction-Following Robustness of Large Lan...

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Li, Zekun Peng, Baolin He, Pengcheng Yan, Xifeng University of California Santa Barbara United States Microsoft Research Redmond United States Zoom United States

ISBN: (纸本)9798891761643

Large language Models (LLMs) have demonstrated exceptional proficiency in instruction-following, making them increasingly integral to various applications. However, this capability introduces the risk of prompt injection attacks, where malicious instructions are embedded in the input to trigger unintended actions or content. Understanding the robustness of LLMs against such attacks is critical for ensuring their safe deployment. In this work, we establish a benchmark to evaluate the robustness of instruction-following LLMs against prompt injection attacks, assessing their ability to discern which instructions to follow and which to disregard. Through extensive experiments with leading instruction-following LLMs, we reveal significant vulnerabilities, particularly in models that mis-follow injected instructions. Our results show that certain models are excessively inclined to prioritize embedded instructions in prompts, often focusing on the latter parts of the prompt without fully understanding the overall context. Conversely, models that exhibit stronger contextual understanding and instruction-following capabilities tend to be more easily compromised by injected instructions. These findings highlight the need to balance improving LLMs' instruction-following abilities with enhancing their overall comprehension of prompts, to prevent mis-following inappropriate instructions. We hope our analysis provides valuable insights into these vulnerabilities, contributing to the development of more robust solutions in the future. © 2024 Association for Computational Linguistics.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

In-Context Compositional Generalization for Large Vision-language Models

In-Context Compositional Generalization for Large Vision-Lan...

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Li, Chuanhao Jing, Chenchen Li, Zhen Zhai, Mingliang Wu, Yuwei Jia, Yunde Beijing Key Laboratory of Intelligent Information Technology School of Computer Science & Technology Beijing Institute of Technology China Guangdong Laboratory of Machine Perception and Intelligent Computing Shenzhen MSU-BIT University China School of Computer Science Zhejiang University Hangzhou China

ISBN: (纸本)9798891761643

Recent work has revealed that in-context learning for large language models exhibits compositional generalization capacity, which can be enhanced by selecting in-context demonstrations similar to test cases to provide contextual information. However, how to exhibit in-context compositional generalization (ICCG) of large vision-language models (LVLMs) is non-trival. Due to the inherent asymmetry between visual and linguistic modalities, ICCG in LVLMs faces an inevitable challenge-redundant information on the visual modality. The redundant information affects in-context learning from two aspects: (1) Similarity calculation may be dominated by redundant information, resulting in sub-optimal demonstration selection. (2) Redundant information in in-context demonstrations brings misleading contextual information to in-context learning. To alleviate these problems, we propose a demonstration selection method to achieve ICCG for LVLMs, by considering two key factors of demonstrations: content and structure, from a multimodal perspective. Specifically, we design a diversity-coverage-based matching score to select demonstrations with maximum coverage, and avoid selecting demonstrations with redundant information via their content redundancy and structural complexity. We build a GQA-ICCG dataset to simulate the ICCG setting, and conduct experiments on GQA-ICCG and the VQA v2 dataset. Experimental results demonstrate the effectiveness of our method. © 2024 Association for Computational Linguistics.

关键词： Visual languages

来源：评论

学校读者我要写书评

暂无评论

Symbolic Working Memory Enhances language Models for Complex Rule Application

Symbolic Working Memory Enhances Language Models for Complex...

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Wang, Siyuan Wei, Zhongyu Choi, Yejin Ren, Xiang University of Southern California United States Fudan University China University of Washington United States Allen Institute for Artificial Intelligence United States

ISBN: (纸本)9798891761643

Large language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning involving a series of rule application steps, especially when rules are presented non-sequentially. Our preliminary analysis shows that while LLMs excel in single-step rule application, their performance drops significantly in multi-step scenarios due to the challenge in rule grounding. It requires anchoring the applicable rule and supporting facts at each step, amidst multiple input rules, facts, and inferred facts. To address this, we propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. The memory stores facts and rules in both natural language and symbolic forms, enabling precise tracking. Utilizing this memory, our framework iteratively performs symbolic rule grounding and LLM-based rule implementation. The former matches predicates and variables of symbolic rules and facts to ground applicable rules at each step. Experiments indicate our framework's effectiveness in rule application and its robustness across various steps and settings. © 2024 Association for Computational Linguistics.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Formality is Favored: Unraveling the Learning Preferences of Large language Models on Data with Conflicting Knowledge

Formality is Favored: Unraveling the Learning Preferences of...

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Li, Jiahuan Cao, Yiqing Huang, Shujian Chen, Jiajun National Key Laboratory for Novel Software Technology Nanjing University China

ISBN: (纸本)9798891761643

Having been trained on massive pretraining data, large language models have shown excellent performance on many knowledge-intensive tasks. However, pretraining data tends to contain misleading and even conflicting information, and it is intriguing to understand how LLMs handle these noisy data during training. In this study, we systematically analyze LLMs' learning preferences for data with conflicting knowledge. We find that pretrained LLMs establish learning preferences similar to humans, i.e., preferences towards formal texts and texts with fewer spelling errors, resulting in faster learning and more favorable treatment of knowledge in data with such features when facing conflicts. This finding is generalizable across models and languages and is more evident in larger models. An in-depth analysis reveals that LLMs tend to trust data with features that signify consistency with the majority of data, and it is possible to instill new preferences and erase old ones by manipulating the degree of consistency with the majority data. © 2024 Association for Computational Linguistics.

关键词： Contrastive Learning

来源：评论

学校读者我要写书评

暂无评论

How do Large language Models Learn In-Context? Query and Key Matrices of In-Context Heads are Two Towers for Metric Learning

How do Large Language Models Learn In-Context? Query and Key...

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Yu, Zeping Ananiadou, Sophia Department of Computer Science National Centre for Text Mining The University of Manchester United Kingdom

ISBN: (纸本)9798891761643

We investigate the mechanism of in-context learning (ICL) on sentence classification tasks with semantically-unrelated labels ("foo"/"bar"). We find intervening in only 1% heads (named "in-context heads") significantly affects ICL accuracy from 87.6% to 24.4%. To understand this phenomenon, we analyze the value-output vectors in these heads and discover that the vectors at each label position contain substantial information about the corresponding labels. Furthermore, we observe that the prediction shift from "foo" to "bar" is due to the respective reduction and increase in these heads' attention scores at "foo" and "bar" positions. Therefore, we propose a hypothesis for ICL: in in-context heads, the value-output matrices extract label features, while the query-key matrices compute the similarity between the features at the last position and those at each label position. The query and key matrices can be considered as two towers that learn the similarity metric between the last position's features and each demonstration at label positions. Using this hypothesis, we explain the majority label bias and recency bias in ICL and propose two methods to reduce these biases by 22% and 17%, respectively. © 2024 Association for Computational Linguistics.

关键词： Contrastive Learning

来源：评论

学校读者我要写书评

暂无评论

Relation labeling in product knowledge graphs with large language models for e-commerce

引用

INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS 2024年第12期15卷 5725-5743页

作者： Chen, Jiao Ma, Luyi Li, Xiaohan Xu, Jianpeng Cho, Jason H. D. Nag, Kaushiki Korpeoglu, Evren Kumar, Sushant Achan, Kannan Walmart Global Tech Personalizat Team Sunnyvale CA 94086 USA

Product Knowledge Graphs (PKGs) play a crucial role in enhancing e-commerce system performance by providing structured information about entities and their relationships, such as complementary or substitutable relations between products or product types, which can be utilized in recommender systems. However, relation labeling in PKGs remains a challenging task due to the dynamic nature of e-commerce domains and the associated cost of human labor. Recently, breakthroughs in Large language Models (LLMs) have shown surprising results in numerous natural language processing tasks, especially in the in-context learning (ICL). In this paper, we conduct an empirical study of LLMs for relation labeling in e-commerce PKGs, investigating their powerful learning capabilities in natural language and effectiveness in predicting relations between product types with few-shot in-context learning. We evaluate the performance of various LLMs, including PaLM-2, GPT-3.5, and Llama-2, on benchmark datasets for e-commerce relation labeling tasks. We use different prompt engineering techniques to examine their impact on model performance. Our results show that LLMs can achieve competitive performance compared to human labelers using just 1-5 labeled examples per relation. We also illustrate the bias issues in LLMs towards minority ethnic groups. Additionally, we show that LLMs significantly outperform existing KG completion models or classification methods in relation labeling for e-commerce KGs and exhibit performance strong enough to replace human labeling. Beyond empirical investigations, we also carry out a theoretical analysis to explain the superior capability of LLMs in few-shot ICL by comparing it with kernel regression.

关键词： Product Relation Knowledge Graph Large language Model Few-shot learning Recommendation E-commerce

来源：评论

学校读者我要写书评

暂无评论

CompoundPiece: Evaluating and Improving Decompounding Performance of language Models

CompoundPiece: Evaluating and Improving Decompounding Perfor...

引用

conference on empirical methods in natural language processing (EMNLP)

作者： Minixhofer, Benjamin Pfeiffer, Jonas Vulic, Ivan Univ Cambridge Cambridge England Google DeepMind London England

ISBN: (纸本)9798891760608

While many languages possess processes of joining two or more words to create compound words, previous studies have been typically limited only to languages with excessively productive compound formation (e.g., German, Dutch) and there is no public dataset containing compound and non-compound words across a large number of languages. In this work, we systematically study decompounding, the task of splitting compound words into their constituents, at a wide scale. We first address the data gap by introducing a dataset of 255k compound and non-compound words across 56 diverse languages obtained from Wiktionary. We then use this dataset to evaluate an array of Large language Models (LLMs) on the decompounding task. We find that LLMs perform poorly, especially on words which are tokenized unfavorably by subword tokenization. We thus introduce a novel methodology to train dedicated models for decompounding. The proposed two-stage procedure relies on a fully self-supervised objective in the first stage, while the second, supervised learning stage optionally fine-tunes the model on the annotated Wiktionary data. Our self-supervised models outperform the prior best unsupervised decompounding models by 13.9% accuracy on average. Our fine-tuned models outperform all prior (language-specific) decompounding tools. Furthermore, we use our models to leverage decompounding during the creation of a subword tokenizer, which we refer to as CompoundPiece. CompoundPiece tokenizes compound words more favorably on average, leading to improved performance on decompounding over an otherwise equivalent model using SentencePiece tokenization.

关键词： Large datasets

来源：评论

学校读者我要写书评

暂无评论

ShadowLLM: Predictor-based Contextual Sparsity for Large language Models

ShadowLLM: Predictor-based Contextual Sparsity for Large Lan...

引用

2024 conference on empirical methods in natural language processing, EMNLP 2024

作者： Akhauri, Yash AbouElhamayed, Ahmed F. Dotzel, Jordan Zhang, Zhiru Rush, Alexander M. Huda, Safeen Abdelfattah, Mohamed S. Cornell University United States Google United States

ISBN: (纸本)9798891761643

The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and *** sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade *** work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation *** this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in *** develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior *** addition, ShadowLLM achieves up to a 20% speedup over the state-of-the-art DejaVu *** enhancements are validated on Llama-2 and OPT models with up to 30 billion *** code is available at ShadowLLM. © 2024 Association for Computational Linguistics.

关键词： Neurons

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：