检索结果-内蒙古大学图书馆

arXiv 2024年

作者： Lyu, Yuanjie Li, Zhiyu Niu, Simin Xiong, Feiyu Tang, Bo Wang, Wenjin Wu, Hao Liu, Huanyong Xu, Tong Chen, Enhong University of Science and Technology of China Hefei China Institute for Advanced Algorithms Research Shanghai China Renmin University of China Beijing China 360 AI Research Institute Beijing China

Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate "hallucinated" content. However, evaluating RAG systems is a challenge. Most benchmarks focus primarily on question answering applications, neglecting other potential scenarios where RAG could be beneficial. Accordingly, in the experiments, these benchmarks often assess only the LLM components of the RAG pipeline or the retriever in knowledge-intensive scenarios, overlooking the impact of external knowledge base construction and the retrieval component on the entire RAG pipeline in non-knowledge-intensive scenarios. To address these issues, this paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios. Specifically, we refer to the CRUD actions that describe interactions between users and knowledge bases, and also categorize the range of RAG applications into four distinct types—Create, Read, Update, and Delete (CRUD). "Create" refers to scenarios requiring the generation of original, varied content. "Read" involves responding to intricate questions in knowledge-intensive situations. "Update" focuses on revising and rectifying inaccuracies or inconsistencies in pre-existing texts. "Delete" pertains to the task of summarizing extensive texts into more concise forms. For each of these CRUD categories, we have developed different datasets to evaluate the performance of RAG systems. We also analyze the effects of various components of the RAG system, such as the retriever, context length, knowledge base construction, and LLM. Finally, we provide useful insights for optimizing the RAG technology for different scenarios. Copyright © 2024, The Authors. All rights reserved.

关键词： Benchmarking

来源：评论

学校读者我要写书评

暂无评论

HiBid: A Cross-Channel Constrained Bidding System with Budget Allocation by Hierarchical Offline Deep Reinforcement Learning

arXiv

引用

arXiv 2023年

作者： Wang, Hao Tang, Bo Liu, Chi Harold Mao, Shangqin Zhou, Jiahong Dai, Zipeng Sun, Yaqi Xie, Qianlong Wang, Xingxing Wang, Dong School of Computer Science and Technology Beijing Institute of Technology Beijing100081 China Meituan and Institute for Advanced Algorithms Research Shanghai China Meituan Beijing China

Online display advertising platforms service numerous advertisers by providing real-time bidding (RTB) for the scale of billions of ad requests every day. The bidding strategy handles ad requests cross multiple channels to maximize the number of clicks under the set financial constraints, i.e., total budget and cost-per-click (CPC), etc. Different from existing works mainly focusing on single channel bidding, we explicitly consider cross-channel constrained bidding with budget allocation. Specifically, we propose a hierarchical offline deep reinforcement learning (DRL) framework called "HiBid", consisted of a high-level planner equipped with auxiliary loss for non-competitive budget allocation, and a data augmentation enhanced low-level executor for adaptive bidding strategy in response to allocated budgets. Additionally, a CPC-guided action selection mechanism is introduced to satisfy the cross-channel CPC constraint. Through extensive experiments on both the large-scale log data and online A/B testing, we confirm that HiBid outperforms six baselines in terms of the number of clicks, CPC satisfactory ratio, and return-on-investment (ROI). We also deploy HiBid on Meituan advertising platform to already service tens of thousands of advertisers every day. Copyright © 2023, The Authors. All rights reserved.

关键词： Budget control

来源：评论

学校读者我要写书评

暂无评论

UBENCH: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions

arXiv

引用

arXiv 2024年

作者： Wang, Xunzhi Zhang, Zhuowei Li, Qiongyu Chen, Gaonan Hu, Mengting Li, Zhiyu Luo, Bitong Gao, Hang Han, Zhixin Wang, Haotian College of Software Nankai University China Institute for Advanced Algorithms Research Shanghai China College of Artificial Intelligence Tianjin University of Science and Technology China

The rapid development of large language models (LLMs) has shown promising practical results. However, their low interpretability often leads to errors in unforeseen circumstances, limiting their utility. Many works have focused on creating comprehensive evaluation systems, but previous benchmarks have primarily assessed problem-solving abilities while neglecting the response's uncertainty, which may result in unreliability. Recent methods for measuring LLM reliability are resource-intensive and unable to test black-box models. To address this, we propose UBENCH, a comprehensive benchmark for evaluating LLM reliability. UBENCH includes 3,978 multiple-choice questions covering knowledge, language, understanding, and reasoning abilities. Experimental results show that UBENCH has achieved state-of-the-art performance, while its single-sampling method significantly saves computational resources compared to baseline methods that require multiple samplings. Additionally, based on UBENCH, we evaluate the reliability of 15 popular LLMs, finding GLM4 to be the most outstanding, closely followed by GPT-4. We also explore the impact of Chain-of-Thought prompts, role-playing prompts, option order, and temperature on LLM reliability, analyzing the varying effects on different LLMs. Copyright © 2024, The Authors. All rights reserved.

关键词： Reliability

来源：评论

学校读者我要写书评

暂无评论

NC-ALG: Graph-Based Active Learning under Noisy Crowd 40

NC-ALG: Graph-Based Active Learning under Noisy Crowd

引用

40th IEEE International Conference on Data Engineering, ICDE 2024

作者： Zhang, Wentao Wang, Yexin You, Zhenbang Li, Yang Cao, Gang Yang, Zhi Cui, Bin Center for Machine Learning Research Peking University China Key Lab of High Confidence Software Technologies Peking University China Institute of Advanced Algorithms Research Shanghai China Institute of Computational Social Science Peking University Qingdao China National Engineering Labratory for Big Data Analytics and Applications China TEG Tencent Inc. Department of Data Platform China Beijing Academy of Artificial Intelligence China

ISBN: (纸本)9798350317152

Graph Neural Networks (GNNs) have achieved great success in various data mining tasks but they heavily rely on a large number of annotated nodes, requiring considerable human efforts. Despite the effectiveness of existing GNN-based Active Learning (AL) methods, they assume that the annotated labels are always correct, which is contradictory to the error-prone labeling process in a practical crowdsourcing environment. Besides, due to this impractical assumption, existing works only focus on optimizing the node selection in AL but neglect optimizing the labeling process. Therefore, we present NC-ALG, the first GNN-based AL framework that optimizes both the node selection and node labeling process under a noisy crowd. For node selection, NC-ALG introduces a new measurement to model influence reliability and an effective influence maximization objective to select nodes. For node labeling, NC-ALG significantly reduces the labeling cost by considering the model-predicted labels and the labels of mirror nodes. To the best of our knowledge, this is the first attempt to consider GNN-based AL under the practical noisy crowd. Empirical studies on public datasets demonstrate that NC-ALG significantly outperforms existing methods in terms labeling efficiency. Notably, it only takes NC-ALG one-third of the labeling budget that the competitive baseline GRAIN needs to achieve an accuracy of 70.7 % on PubMed. © 2024 IEEE.

关键词： Crowdsourcing

来源：评论

学校读者我要写书评

暂无评论

Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing 38

Initialization is Critical to Whether Transformers Fit Compo...

引用

38th Conference on Neural Information Processing Systems, NeurIPS 2024

作者： Zhang, Zhongwang Lin, Pengxiao Wang, Zhiwei Zhang, Yaoyu Xu, Zhi-Qin John Institute of Natural Sciences MOE-LSC Shanghai Jiao Tong University China School of Mathematical Sciences Shanghai Jiao Tong University China Key Laboratory of Marine Intelligent Equipment and System Ministry of Education China Shanghai Seres Information Technology Co. Ltd. Shanghai China Center for LLM Institute for Advanced Algorithms Research Shanghai China

Transformers have shown impressive capabilities across various tasks, but their performance on compositional problems remains a topic of debate. In this work, we investigate the mechanisms of how transformers behave on unseen compositional tasks. We discover that the parameter initialization scale plays a critical role in determining whether the model learns inferential (reasoning-based) solutions, which capture the underlying compositional primitives, or symmetric (memory-based) solutions, which simply memorize mappings without understanding the compositional structure. By analyzing the information flow and vector representations within the model, we reveal the distinct mechanisms underlying these solution types. We further find that inferential (reasoning-based) solutions exhibit low complexity bias, which we hypothesize is a key factor enabling them to learn individual mappings for single anchors. We validate our conclusions on various real-world datasets. Our findings provide valuable insights into the role of initialization scale in tuning the reasoning and memorizing ability and we propose the initialization rate γ to be a convenient tunable hyper-parameter in common deep learning frameworks, where 1/dγin is the standard deviation of parameters of the layer with din input neurons. © 2024 Neural information processing systems foundation. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

FastMem: Fast Memorization of Prompt Improves Context Awareness of Large Language Models

arXiv

引用

arXiv 2024年

作者： Zhu, Junyi Liu, Shuochen Yu, Yu Tang, Bo Yan, Yibo Li, Zhiyu Xiong, Feiyu Xu, Tong Blaschko, Matthew B. ESAT-PSI KU Leuven Belgium University of Science and Technology of China China Institute for Advanced Algorithms Research Shanghai China National University of Singapore Singapore

Large language models (LLMs) excel in generating coherent text, but they often struggle with context awareness, leading to inaccuracies in tasks requiring faithful adherence to provided information. We introduce FastMem, a novel method designed to enhance instruction fine-tuned LLMs’ context awareness through fast memorization of the prompt. FastMem maximizes the likelihood of the prompt before inference by updating only the last Feed-Forward Network (FFN) module. This targeted approach ensures efficient optimization without overfitting, significantly improving the model’s ability to comprehend and accurately follow the context. Our experiments demonstrate substantial gains in reading comprehension, text summarization and adherence to output structures. For instance, FastMem improves the accuracy of Llama 3-8B-Inst on the NQ-SWAP dataset from 59.1% to 71.6%, and reduces the output structure failure rate of Qwen 1.5-4B-Chat from 34.9% to 25.5%. Extensive experimental results highlight FastMem’s potential to offer a robust solution to enhance the reliability and accuracy of LLMs in various applications. Our code is available at: https://***/IAAR-Shanghai/FastMem. Copyright © 2024, The Authors. All rights reserved.

关键词： Failure rate

来源：评论

学校读者我要写书评

暂无评论

Memory3: Language Modeling with Explicit Memory

arXiv

引用

arXiv 2024年

作者： Yang, Hongkang Lin, Zehao Wang, Wenjin Wu, Hao Li, Zhiyu Tang, Bo Wei, Wenqiang Wang, Jinbo Tang, Zeyun Song, Shichao Xi, Chenyang Yu, Yu Chen, Kai Xiong, Feiyu Tang, Linpeng Weinan, E. Center for LLM Institute for Advanced Algorithms Research Shanghai China Moqi Inc China Center for Machine Learning Research Peking University China School of Mathematical Sciences Peking University AI for Science Institute China

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named Memory3, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory *** Codes 68T50 Copyright © 2024, The Authors. All rights reserved.

关键词： Digital storage

来源：评论

学校读者我要写书评

暂无评论

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy

arXiv

引用

arXiv 2024年

作者： Zhu, Yu Sun, Chuxiong Yang, Wenfei Wei, Wenqiang Tang, Bo Zhang, Tianzhu Li, Zhiyu Zhang, Shifeng Xiong, Feiyu Hu, Jie Yang, Mingchuan University of Science and Technology of China Hefei China Deep Space Exploration Laboratory China Institute for Advanced Algorithms Research Shanghai China Research Institute of China Telecom China Sangfor Technologies Inc. China

Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1% of the training parameters of other methods. Copyright © 2024, The Authors. All rights reserved.

关键词： Reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation

arXiv

引用

arXiv 2023年

作者： Liang, Xun Song, Shichao Niu, Simin Li, Zhiyu Xiong, Feiyu Tang, Bo Wang, Yezhaohui He, Dawei Cheng, Peng Wang, Zhonghao Deng, Haiying School of Information Renmin University of China Beijing China Institute for Advanced Algorithms Research Shanghai China State Key Laboratory of Media Convergence Production Technology and Systems Beijing China

Large language models (LLMs) produce hallucinated text, compromising their practical utility in professional contexts. To assess the reliability of LLMs, numerous initiatives have developed benchmark evaluations for hallucination phenomena. However, they often employ constrained generation techniques to produce the evaluation dataset due to cost and time limitations. For instance, this may involve employing directed hallucination induction or deliberately modifying authentic text to generate hallucinations. These are not congruent with the unrestricted text generation demanded by real-world applications. Furthermore, a well-established Chinese-language dataset dedicated to the evaluation of hallucinations is presently lacking. Consequently, we have developed an Unconstrained Hallucination Generation Evaluation (UHGEval) benchmark, containing hallucinations generated by LLMs with minimal restrictions1. Concurrently, we have established a comprehensive benchmark evaluation framework to aid subsequent researchers in undertaking scalable and reproducible experiments. We have also evaluated prominent Chinese LLMs and the GPT series models to derive insights regarding hallucination. Copyright © 2023, The Authors. All rights reserved.

关键词： Petroleum reservoir evaluation

来源：评论

学校读者我要写书评

暂无评论

NewsBench: A Systematic Evaluation Framework for Assessing Editorial Capabilities of Large Language Models in Chinese Journalism

arXiv

引用

arXiv 2024年

作者： Li, Miao Chen, Ming-Bin Tang, Bo Hou, Shengbin Wang, Pengyu Deng, Haiying Li, Zhiyu Xiong, Feiyu Mao, Keming Cheng, Peng Luo, Yi School of Computing and Information Systems The University of Melbourne Australia Institute for Advanced Algorithms Research China Northeastern University China State Key Laboratory of Media Convergence Production Technology and Systems China

We present NewsBench, a novel evaluation framework to systematically assess the capabilities of Large Language Models (LLMs) for editorial capabilities in Chinese journalism. Our constructed benchmark dataset is focused on four facets of writing proficiency and six facets of safety adherence, and it comprises manually and carefully designed 1,267 test samples in the types of multiple choice questions and short answer questions for five editorial tasks in 24 news domains. To measure performances, we propose different GPT-4 based automatic evaluation protocols to assess LLM generations for short answer questions in terms of writing proficiency and safety adherence, and both are validated by the high correlations with human evaluations. Based on the systematic evaluation framework, we conduct a comprehensive analysis of eleven popular LLMs which can handle Chinese. The experimental results highlight GPT-4 and ERNIE Bot as top performers, yet reveal a relative deficiency in journalistic safety adherence in creative writing tasks. Our findings also underscore the need for enhanced ethical guidance in machine-generated journalistic content, marking a step forward in aligning LLMs with journalistic standards and safety considerations. The evaluation framework and experimental results are expected to provide an in-depth understanding of the editorial capabilities of LLMs and speed up the development of LLMs in journalism.1 © 2024, CC BY.

关键词： Statistical tests

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：