检索结果-内蒙古大学图书馆

arXiv 2024年

作者： Yu, Qingchen Zheng, Zifan Song, Shichao Li, Zhiyu Xiong, Feiyu Tang, Bo Chen, Ding Institute for Advanced Algorithms Research Shanghai China Renmin University of China China

The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. As evaluation frameworks commonly use Regular Expression (RegEx) for answer extraction, models may adjust their responses to fit formats easily handled by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. Furthermore, recent studies proposing fine-tuned LLMs as judge models for automated evaluation face challenges in terms of generalization ability and fairness. This paper comprehensively analyzes the entire LLM evaluation chain and demonstrates that optimizing the key answer extraction module improves extraction accuracy and enhances evaluation reliability. Our findings suggest that improving the key answer extraction module can lead to higher judgment accuracy and improved evaluation efficiency compared to the judge models. To address these issues, we propose xFinder, a novel evaluator for answer extraction and matching in LLM evaluation. As part of this process, we create a specialized dataset, the Key Answer Finder (KAF) dataset, to ensure effective model training and evaluation. Generalization tests and real-world evaluations show that the smallest xFinder model, with only 500 million parameters, achieves an average extraction accuracy of 93.42%. In contrast, RegEx accuracy in the best evaluation framework is 74.38%. The final judgment accuracy of xFinder reaches 97.61%, outperforming existing evaluation frameworks and judge models. All resources for xFinder are available at https://***/IAAR-Shanghai/xFinder. Copyright © 2024, The Authors. All rights reserved.

关键词： Digital elevation model

来源：评论

学校读者我要写书评

暂无评论

META-CHUNKING: LEARNING EFFICIENT TEXT SEGMENTATION VIA LOGICAL PERCEPTION

arXiv

引用

arXiv 2024年

作者： Zhao, Jihao Ji, Zhiyuan Feng, Yuchen Qi, Pengnian Niu, Simin Tang, Bo Xiong, Feiyu Li, Zhiyu Renmin University of China China Institute for Advanced Algorithms Research Shanghai China

Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline, which impacts the quality of knowledge-intensive tasks. This paper introduces the concept of Meta-Chunking, which refers to a granularity between sentences and paragraphs, consisting of a collection of sentences within a paragraph that have deep linguistic logical connections. To implement Meta-Chunking, we designed Perplexity (PPL) Chunking, which balances performance and speed, and precisely identifies the boundaries of text chunks by analyzing the characteristics of context perplexity distribution. Additionally, considering the inherent complexity of different texts, we propose a strategy that combines PPL Chunking with dynamic merging to achieve a balance between fine-grained and coarse-grained text chunking. Experiments conducted on eleven datasets demonstrate that Meta-Chunking can more efficiently improve the performance of single-hop and multi-hop question answering based on RAG. For instance, on the 2Wiki-MultihopQA dataset, it outperforms similarity chunking by 1.32 while only consuming 45.8% of the time. Furthermore, through the analysis of models of various scales and types, we observed that PPL Chunking exhibits notable flexibility and adaptability. Our code is available at https://***/IAAR-Shanghai/MetaChunking. Copyright © 2024, The Authors. All rights reserved.

关键词： Linguistics

来源：评论

学校读者我要写书评

暂无评论

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios

arXiv

引用

arXiv 2024年

作者： Ruan, Jiacheng Yuan, Wenzhen Lin, Zehao Liao, Ning Li, Zhiyu Xiong, Feiyu Liu, Ting Fu, Yuzhuo Shanghai Jiao Tong University China Institute for Advanced Algorithms Research Shanghai China

Large visual-language models (LVLMs) have achieved great success in multiple applications. However, they still encounter challenges in complex scenes, especially those involving camouflaged objects. This is primarily due to the lack of samples related to camouflaged scenes in the training dataset. To mitigate this issue, we construct the MM-CamObj dataset for the first time, comprising two subsets: CamObj-Align and CamObj-Instruct. Specifically, CamObj-Align contains 11,363 image-text pairs, and it is designed for VL alignment and injecting rich knowledge of camouflaged scenes into LVLMs. CamObj-Instruct is collected for fine-tuning the LVLMs with improved instruction-following capabilities, and it includes 11,363 images and 68,849 conversations with diverse instructions. Based on the MM-CamObj dataset, we propose the CamObj-Llava, an LVLM specifically designed for addressing tasks in camouflaged scenes. To facilitate our model’s effective acquisition of knowledge about camouflaged objects and scenes, we introduce a curriculum learning strategy with six distinct modes. Additionally, we construct the CamObj-Bench to evaluate the existing LVLMs’ capabilities of understanding, recognition, localization and count in camouflage scenes. This benchmark includes 600 images and 7 tasks, with a total of 9,449 questions. Extensive experiments are conducted on the CamObj-Bench with CamObj-Llava, 8 existing open-source and 3 closed-source LVLMs. Surprisingly, the results indicate that our model achieves a 25.84% improvement in 4 out of 7 tasks compared to GPT-4o. Code and datasets will be available at https://***/JCruan519/MM-CamObj. Copyright © 2024, The Authors. All rights reserved.

关键词： Visual languages

来源：评论

学校读者我要写书评

暂无评论

FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion

arXiv

引用

arXiv 2024年

作者： Ruan, Jiacheng Yang, Yebin Lin, Zehao Feng, Yuchen Xiong, Feiyu Tang, Zeyun Li, Zhiyu Shanghai Jiao Tong University China Institute for Advanced Algorithms Research Shanghai China

Benefiting from the revolutionary advances in large language models (LLMs) and foundational vision models, large vision-language models (LVLMs) have also made significant progress. However, current benchmarks focus on tasks that evaluating only a single aspect of LVLM capabilities (e.g., recognition, detection, understanding). These tasks fail to fully demonstrate LVLMs’ potential in complex application scenarios. To comprehensively assess the performance of existing LVLMs, we propose a more challenging task called the Flow Text with Image Insertion task (FTII). This task requires LVLMs to simultaneously possess outstanding abilities in image comprehension, instruction understanding, and long-text interpretation. Specifically, given several text paragraphs and a set of candidate images, as the text paragraphs accumulate, the LVLMs are required to select the most suitable image from the candidates to insert after the corresponding paragraph. Constructing a benchmark for such a task is highly challenging, particularly in determining the sequence of flowing text and images. To address this challenge, we turn to professional news reports, which naturally contain a gold standard for image-text sequences. Based on this, we introduce the Flow Text with Image Insertion Benchmark (FTII-Bench), which includes 318 high-quality Chinese image-text news articles and 307 high-quality English image-text news articles, covering 10 different news domains. Using these 625 high-quality articles, we construct problems of two different types with multiple levels of difficulty. Furthermore, we establish two different evaluation pipelines based on the CLIP model and existing LVLMs. We evaluate 9 open-source and 2 closed-source LVLMs as well as 2 CLIP-based models. Results indicate that even the most advanced models (e.g., GPT-4o) face significant challenges when tackling the FTII task. The code and data will be available at https://***/IAAR-Shanghai/FTIIBench. Copyright © 2024, The

关键词： Open systems

来源：评论

学校读者我要写书评

暂无评论

HRDE: Retrieval-Augmented Large Language Models for Chinese Health Rumor Detection and Explainability

arXiv

引用

arXiv 2024年

作者： Chen, Yanfang Chen, Ding Song, Shichao Niu, Simin Wang, Hanyu Tang, Zeyun Xiong, Feiyu Li, Zhiyu Renmin University of China Beijing China Institute for Advanced Algorithms Research Shanghai China

As people increasingly prioritize their health, the speed and breadth of health information dissemination on the internet have also grown. At the same time, the presence of false health information (health rumors) intermingled with genuine content poses a significant potential threat to public health. However, current research on Chinese health rumors still lacks a large-scale, public, and open-source dataset of health rumor information, as well as effective and reliable rumor detection methods. This paper addresses this gap by constructing a dataset containing 1.12 million health-related rumors (HealthRCN) through web scraping of common health-related questions and a series of data processing steps. HealthRCN is the largest known dataset of Chinese health information rumors to date. Based on this dataset, we propose retrieval-augmented large language models for Chinese health rumor detection and explainability (HRDE 1 2). This model leverages retrieved relevant information to accurately determine whether the input health information is a rumor and provides explanatory responses, effectively aiding users in verifying the authenticity of health information. In evaluation experiments, we compared multiple models and found that HRDE outperformed them all, including GPT-4-1106-Preview, in rumor detection accuracy and answer quality. HRDE achieved an average accuracy of 91.04% and an F1 score of 91.58%. Copyright © 2024, The Authors. All rights reserved.

关键词： Health risks

来源：评论

学校读者我要写书评

暂无评论

QAEncoder: Towards Aligned Representation Learning in Question Answering System

arXiv

引用

arXiv 2024年

作者： Wang, Zhengren Yu, Qinhan Wei, Shida Li, Zhiyu Xiong, Feiyu Wang, Xiaoxing Niu, Simin Liang, Hao Zhang, Wentao Peking University China Center for LLM Institute for Advanced Algorithms Research Shanghai China

Modern QA systems entail retrieval-augmented generation (RAG) for accurate and trustworthy responses. However, the inherent gap between user queries and relevant documents hinders precise matching. Motivated by our conical distribution hypothesis, which posits that potential queries and documents form a cone-like structure in the embedding space, we introduce QAEncoder, a training-free approach to bridge this gap. Specifically, QAEncoder estimates the expectation of potential queries in the embedding space as a robust surrogate for the document embedding, and attaches document fingerprints to effectively distinguish these embeddings. Extensive experiments on fourteen embedding models across six languages and eight datasets validate QAEncoder’s alignment capability, which offers a plug-and-play solution that seamlessly integrates with existing RAG architectures and training-based methods. The code is available at https://***/IAAR-Shanghai/QAEncoder. Copyright © 2024, The Authors. All rights reserved.

关键词： Embeddings

来源：评论

学校读者我要写书评

暂无评论

Controllable Text Generation for Large Language Models: A Survey

arXiv

引用

arXiv 2024年

作者： Liang, Xun Wang, Hanyu Wang, Yezhaohui Song, Shichao Yang, Jiawei Niu, Simin Hu, Jie Liu, Dan Yao, Shunyu Xiong, Feiyu Li, Zhiyu Renmin University of China China Institute for Advanced Algorithms Research Shanghai China China Telecom Research Institute China

In Natural Language Processing (NLP), Large Language Models (LLMs) have demonstrated high text generation quality. However, in real-world applications, LLMs must meet increasingly complex requirements. Beyond avoiding misleading or inappropriate content, LLMs are also expected to cater to specific user needs, such as imitating particular writing styles or generating text with poetic richness. These varied demands have driven the development of Controllable Text Generation (CTG) techniques, which ensure that outputs adhere to predefined control conditions—such as safety, sentiment, thematic consistency, and linguistic style—while maintaining high standards of helpfulness, fluency, and diversity. This paper systematically reviews the latest advancements in CTG for LLMs, offering a comprehensive definition of its core concepts and clarifying the requirements for control conditions and text quality. We categorize CTG tasks into two primary types: content control and attribute control. The key methods are discussed, including model retraining, fine-tuning, reinforcement learning, prompt engineering, latent space manipulation, and decoding-time intervention. We analyze each method’s characteristics, advantages, and limitations, providing nuanced insights for achieving generation control. Additionally, we review CTG evaluation methods, summarize its applications across domains, and address key challenges in current research, including reduced fluency and practicality. We also propose several appeals, such as placing greater emphasis on real-world applications in future research. This paper aims to offer valuable guidance to researchers and developers in the field. Our reference list and Chinese version are open-sourced at https://***/IAAR-Shanghai/CTGSurvey. Copyright © 2024, The Authors. All rights reserved.

关键词： Reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

CAT-LLM: Prompting Large Language Models with Text Style Definition for Chinese Article-style Transfer

arXiv

引用

arXiv 2024年

作者： Tao, Zhen Xi, Dinghao Li, Zhiyu Tang, Liumin Xu, Wei School of Information Renmin University of China China Institute for Advanced Algorithms Research Shanghai China

Text style transfer is increasingly prominent in online entertainment and social media. However, existing research mainly concentrates on style transfer within individual English sentences, while ignoring the complexity of long Chinese texts, which limits the wider applicability of style transfer in digital media realm. To bridge this gap, we propose a Chinese Article-style Transfer framework (CAT-LLM), leveraging the capabilities of Large Language Models (LLMs). CAT-LLM incorporates a bespoke, pluggable Text Style Definition (TSD) module aimed at comprehensively analyzing text features in articles, prompting LLMs to efficiently transfer Chinese article-style. The TSD module integrates a series of machine learning algorithms to analyze article-style from both words and sentences levels, thereby aiding LLMs thoroughly grasp the target style without compromising the integrity of the original text. In addition, this module supports dynamic expansion of internal style trees, showcasing robust compatibility and allowing flexible optimization in subsequent research. Moreover, we select five Chinese articles with distinct styles and create five parallel datasets using ChatGPT, enhancing the models’ performance evaluation accuracy and establishing a novel paradigm for evaluating subsequent research on article-style transfer. Extensive experimental results affirm that CAT-LLM outperforms current research in terms of transfer accuracy and content preservation, and has remarkable applicability to various types of LLMs. Source code is available at GitHub. Copyright © 2024, The Authors. All rights reserved.

关键词： Digital storage

来源：评论

学校读者我要写书评

暂无评论

Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

arXiv

引用

arXiv 2018年

作者： El-Nouby, Alaaeldin Sharma, Shikhar Schulz, Hannes Hjelm, Devon Asri, Layla El Kahou, Samira Ebrahimi Bengio, Yoshua Taylor, Graham W. University of Guelph Microsoft Research Montreal Institute for Learning Algorithms Vector Institute for Artificial Intelligence University of Montreal Canadian Institute for Advanced Research

Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from available conditioning information in one step. One practical extension beyond one-step generation is a system that generates an image iteratively, conditioned on ongoing linguistic input or feedback. This is significantly more challenging than one-step generation tasks, as such a system must understand the contents of its generated images with respect to the feedback history, the current feedback, as well as the interactions among concepts present in the feedback history. In this work, we present a recurrent image generation model which takes into account both the generated output up to the current step as well as all past instructions for generation. We show that our model is able to generate the background, add new objects, and apply simple transformations to existing objects. We believe our approach is an important step toward interactive generation. Code and data is available at: https://***/en-us/research/project/generative-neural-visual-artist-geneva/. Copyright © 2018, The Authors. All rights reserved.

关键词： Linguistics

来源：评论

学校读者我要写书评

暂无评论

Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

Tell, Draw, and Repeat: Generating and Modifying Images Base...

引用

International Conference on Computer Vision (ICCV)

作者： Alaaeldin El-Nouby Shikhar Sharma Hannes Schulz R Devon Hjelm Layla El Asri Samira Ebrahimi Kahou Yoshua Bengio Graham Taylor University of Guelph Vector Institute for Artificial Intelligence Microsoft Research Montreal Institute for Learning Algorithms University of Montreal Canadian Institute for Advanced Research

ISBN: (数字)9781728148038

ISBN: (纸本)9781728148045

关键词： Task analysis Linguistics Image generation Generators Visualization History Gallium nitride

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：