检索结果-内蒙古大学图书馆

61st Annual Meeting of the the Association-for-Computational-Linguistics (ACL)

作者： Song, Jongyoon Park, Nohil Hwang, Bongkyu Yung, Jaewoong Joe, Seongho Gwon, Youngjune L. Yoon, Sungroh Seoul Natl Univ Data Sci & AI Lab Seoul South Korea Samsung SDS Seoul South Korea Seoul Natl Univ Deptment ECE Seoul South Korea Seoul Natl Univ Interdisciplinary Program AI Seoul South Korea

ISBN: (纸本)9781959429623

In this study, we analyze the model intrinsic features of a summarization model by varying the fine-tuning objectives and datasets. We fine-tune BART models combining three fine-tuning objectives (negative log-likelihood, unlikelihood, and contrastive loss) and two datasets (CNN/DailyMail and XSum) and provide shuffled or aligned documents to observe changes in the model predictions and intrinsic features. We find that (i) the inductive bias for factual consistency during the fine-tuning procedure depends on both the objectives and datasets, and (ii) summarization models with relatively low factual consistency are more likely to model summaries that are not conditional to the documents. We demonstrate that splitting data based on the unconditional and conditional summary modeling difficulty affects the factual consistency and intrinsic features of the summarization models. Our experimental results highlight the importance of studying the inductive bias during fine-tuning for factual consistency.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models 18

Entity-level Factual Adaptiveness of Fine-tuning based Abstr...

引用

18th Conference of the European-Chapter of the Association-for-Computational-Linguistics (EACL)

作者： Song, Jongyoon Park, Nohil Hwang, Bongkyu Yung, Jaewoong Joe, Seongho Gwon, Youngjune L. Yoon, Sungroh Seoul Natl Univ Data Sci & AI Lab Seoul South Korea Samsung SDS Seoul South Korea Seoul Natl Univ Deptment ECE & Interdisciplinary Program Ai Seoul South Korea Seoul Natl Univ ASRI INMC & AIIS Seoul South Korea

ISBN: (纸本)9798891760882

The Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual data augmentation method where the degree of knowledge conflict within the augmented data can be adjustable. Our experimental results on two pre-trained language models (PEGASUS and BART) and two fine-tuning datasets (XSum and CNN/DailyMail) demonstrate that our method enhances factual adaptiveness while achieving factual consistency on original datasets on par with the contrastive learning baseline.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Large Language Models are Skeptics: False Negative Problem of Input-conflicting Hallucination

arXiv

引用

arXiv 2024年

作者： Song, Jongyoon Yu, Sangwon Yoon, Sungroh Data Science & AI Laboratory Seoul National University Korea Republic of Deptment of ECE Interdisciplinary Program in AI Seoul National University Korea Republic of ASRI INMC AIIS Seoul National University Korea Republic of

In this paper, we identify a new category of bias that induces input-conflicting hallucinations, where large language models (LLMs) generate responses inconsistent with the content of the input context. This issue we have termed the false negative problem refers to the phenomenon where LLMs are predisposed to return negative judgments when assessing the correctness of a statement given the context. In experiments involving pairs of statements that contain the same information but have contradictory factual directions, we observe that LLMs exhibit a bias toward false negatives. Specifically, the model presents greater overconfidence when responding with False. Furthermore, we analyze the relationship between the false negative problem and context and query rewriting and observe that both effectively tackle false negatives in LLMs. Copyright © 2024, The Authors. All rights reserved.

关键词： Computational linguistics

来源：评论

学校读者我要写书评

暂无评论

Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models

arXiv

引用

arXiv 2024年

作者： Song, Jongyoon Park, Nohil Hwang, Bongkyu Yun, Jaewoong Joe, Seongho Gwon, Youngjune L. Yoon, Sungroh Data Science & AI Laboratory Seoul National University Korea Republic of Samsung SDS Korea Republic of Deptment of ECE and Interdisciplinary Program in AI Seoul National University Korea Republic of ASRI INMC AIIS Seoul National University Korea Republic of

Abstractive summarization models often generate factually inconsistent content particularly when the parametric knowledge of the model conflicts with the knowledge in the input document. In this paper, we analyze the robustness of fine-tuning based summarization models to the knowledge conflict, which we call factual adaptiveness. We utilize pre-trained language models to construct evaluation sets and find that factual adaptiveness is not strongly correlated with factual consistency on original datasets. Furthermore, we introduce a controllable counterfactual data augmentation method where the degree of knowledge conflict within the augmented data can be adjustable. Our experimental results on two pre-trained language models (PEGASUS and BART) and two fine-tuning datasets (XSum and CNN/DailyMail) demonstrate that our method enhances factual adaptiveness while achieving factual consistency on original datasets on par with the contrastive learning baseline. Copyright © 2024, The Authors. All rights reserved.

关键词： Machine learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：