CoUDA: Coherence Evaluation via Unified Data Augmentation

作     者：Zhu, Dawei Wu, Wenhao Song, Yifan Zhu, Fangwei Cao, Ziqiang Li, Sujian 

作者机构：School of Computer Science Peking University China National Key Laboratory for Multimedia Information Processing Peking University China Institute of Artificial Intelligence Soochow University China Jiangsu Collaborative Innovation Center for Language Ability Jiangsu Normal University China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

摘      要：Coherence evaluation aims to assess the organization and structure of a discourse, which remains challenging even in the era of large language models. Due to the scarcity of annotated data, data augmentation is commonly used for training coherence evaluation models. However, previous augmentations for this task primarily rely on heuristic rules, lacking designing criteria as guidance. In this paper, we take inspiration from linguistic theory of discourse structure, and propose a data augmentation framework named COUDA. COUDA breaks down discourse coherence into global and local aspects, and designs augmentation strategies for both aspects, respectively. Especially for local coherence, we propose a novel generative strategy for constructing augmentation samples, which involves post-pretraining a generative model and applying two controlling mechanisms to control the difficulty of generated samples. During inference, COUDA also jointly evaluates both global and local aspects to comprehensively assess the overall coherence of a discourse. Extensive experiments in coherence evaluation show that, with only 233M parameters, COUDA achieves state-ofthe-art performance in both pointwise scoring and pairwise ranking tasks, even surpassing recent GPT-3.5 and GPT-4 based metrics. © 2024, CC BY.