检索结果-内蒙古大学图书馆

Target-oriented fine-tuning for zero-resource named entity recognition

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Zhang, Ying Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Zero-resource named entity recognition (NER) severely suffers from data scarcity in a specific domain or language. Most studies on zero-resource NER transfer knowledge from various data by fine-tuning on different auxiliary tasks. However, how to properly select training data and fine-tuning tasks is still an open problem. In this paper, we tackle the problem by transferring knowledge from three aspects, i.e., domain, language and task, and strengthening connections among them. Specifically, we propose four practical guidelines to guide knowledge transfer and task finetuning. Based on these guidelines, we design a target-oriented fine-tuning (TOF) framework to exploit various data from three aspects in a unified training manner. Experimental results on six benchmarks show that our method yields consistent improvements over baselines in both cross-domain and cross-lingual scenarios. Particularly, we achieve new state-of-theart performance on five benchmarks. Copyright © 2021, The Authors. All rights reserved.

关键词： Knowledge management

Modeling bilingual conversational characteristics for neural chat translation

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Liang, Yunlong Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Neural chat translation aims to translate bilingual conversational text, which has a broad application in international exchanges and cooperation. Despite the impressive performance of sentence-level and context-aware Neural Machine Translation (NMT), there still remain challenges to translate bilingual conversational text due to its inherent characteristics such as role preference, dialogue coherence, and translation consistency. In this paper, we aim to promote the translation quality of conversational text by modeling the above properties. Specifically, we design three latent variational modules to learn the distributions of bilingual conversational characteristics. Through sampling from these learned distributions, the latent variables, tailored for role preference, dialogue coherence, and translation consistency, are incorporated into the NMT model for better translation. We evaluate our approach on the benchmark dataset BConTrasT (English-German) and a self-collected bilingual dialogue corpus, named BMELD (English-Chinese). Extensive experiments show that our approach notably boosts the performance over strong baselines by a large margin and significantly surpasses some state-of-the-art context-aware NMT models in terms of BLEU and TER. Additionally, we make the BMELD dataset publicly available for the research community. Copyright © 2021, The Authors. All rights reserved.

关键词： Neural machine translation

Understanding Translationese in Cross-Lingual Summarization

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Wang, Jiaan Meng, Fandong Liang, Yunlong Zhang, Tingyi Xu, Jiarong Li, Zhixu Zhou, Jie Shanghai Key Laboratory of Data Science School of Computer Science Fudan University Shanghai China Pattern Recognition Center WeChat AI Tencent Inc China Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China School of Management Fudan University Shanghai China

Given a document in a source language, cross-lingual summarization (CLS) aims at generating a concise summary in a different target language. Unlike monolingual summarization (MS), naturally occurring source-language documents paired with target-language summaries are rare. To collect large-scale CLS data, existing datasets typically involve translation in their creation. However, the translated text is distinguished from the text originally written in that language, i.e., translationese. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese. Then we systematically investigate how translationese affects CLS model evaluation and performance when it appears in source documents or target summaries. In detail, we find that (1) the translationese in documents or summaries of test sets might lead to the discrepancy between human judgment and automatic evaluation;(2) the translationese in training sets would harm model performance in real-world applications;(3) though machine-translated documents involve translationese, they are very useful for building CLS systems on low-resource languages under specific training strategies. Lastly, we give suggestions for future CLS research including dataset and model developments. We hope that our work could let researchers notice the phenomenon of translationese in CLS and take it into account in the future. Copyright © 2022, The Authors. All rights reserved.

关键词： Large dataset

Towards making the most of dialogue characteristics for neural chat translation

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Liang, Yunlong Zhou, Chulun Meng, Fandong Xu, Jinan Chen, Yufeng Su, Jinsong Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University School of Informatics Xiamen University Pattern Recognition Center WeChat AI Tencent Inc China

Neural Chat Translation (NCT) aims to translate conversational text between speakers of different languages. Despite the promising performance of sentence-level and context-aware neural machine translation models, there still remain limitations in current NCT models because the inherent dialogue characteristics of chat, such as dialogue coherence and speaker personality, are neglected. In this paper, we propose to promote the chat translation by introducing the modeling of dialogue characteristics into the NCT model. To this end, we design four auxiliary tasks including monolingual response generation, cross-lingual response generation, next utterance discrimination, and speaker identification. Together with the main chat translation task, we optimize the NCT model through the training objectives of all these tasks. By this means, the NCT model can be enhanced by capturing the inherent dialogue characteristics, thus generating more coherent and speaker-relevant translations. Comprehensive experiments on four language directions (English⇔German and English⇔Chinese) verify the effectiveness and superiority of the proposed approach. Copyright © 2021, The Authors. All rights reserved.

关键词： Neural machine translation

From Mimicking to Integrating: Knowledge Integration for Pre-Trained Language Models

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Li, Lei Lin, Yankai Ren, Xuancheng Zhao, Guangxiang Li, Peng Zhou, Jie Sun, Xu MOE Key Lab of Computational Linguistics School of Computer Science Peking University China Gaoling School of Artificial Intelligence Renmin University of China Beijing China Beijing Key Laboratory of Big Data Management and Analysis Methods Beijing China Tsinghua University China Pattern Recognition Center WeChat AI Tencent Inc. China

Investigating better ways to reuse the released pre-trained language models (PLMs) can significantly reduce the computational cost and the potential environmental side-effects. This paper explores a novel PLM reuse paradigm, Knowledge Integration (KI). Without human annotations available, KI aims to merge the knowledge from different teacher-PLMs, each of which specializes in a different classification problem, into a versatile student model. To achieve this, we first derive the correlation between virtual golden supervision and teacher predictions. We then design a Model Uncertainty–aware Knowledge Integration (MUKI) framework to recover the golden supervision for the student. Specifically, MUKI adopts Monte-Carlo Dropout to estimate model uncertainty for the supervision integration. An instance-wise re-weighting mechanism based on the margin of uncertainty scores is further incorporated, to deal with the potential conflicting supervision from teachers. Experimental results demonstrate that MUKI achieves substantial improvements over baselines on benchmark datasets. Further analysis shows that MUKI can generalize well for merging teacher models with heterogeneous architectures, and even teachers major in cross-lingual datasets.1 © 2022, CC BY.

关键词： Uncertainty analysis

Prevent the language model from being overconfident in neural machine translation

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Miao, Mengqi Meng, Fandong Liu, Yijin Zhou, Xiao-Hua Zhou, Jie Peking University China Pattern Recognition Center WeChat AI Tencent Inc China Beijing International Center for Mathematical Research National Engineering Lab for Big Data Analysis and Applications Department of Biostatistics Peking University Beijing China

The Neural Machine Translation (NMT) model is essentially a joint language model conditioned on both the source sentence and partial translation. Therefore, the NMT model naturally involves the mechanism of the Language Model (LM) that predicts the next token only based on partial translation. Despite its success, NMT still suffers from the hallucination problem, generating fluent but inadequate translations. The main reason is that NMT pays excessive attention to the partial translation while neglecting the source sentence to some extent, namely overconfidence of the LM. Accordingly, we define the Margin between the NMT and the LM, calculated by subtracting the predicted probability of the LM from that of the NMT model for each token. The Margin is negatively correlated to the overconfidence degree of the LM. Based on the property, we propose a Margin-based Token-level Objective (MTO) and a Margin-based Sentencelevel Objective (MSO) to maximize the Margin for preventing the LM from being overconfident. Experiments on WMT14 English-to-German, WMT19 Chinese-to-English, and WMT14 English-to-French translation tasks demonstrate the effectiveness of our approach, with 1.36, 1.50, and 0.63 BLEU improvements, respectively, compared to the Transformer baseline. The human evaluation further verifies that our approaches improve translation adequacy as well as fluency. Copyright © 2021, The Authors. All rights reserved.

关键词： Neural machine translation

MF-CLIP: Leveraging CLIP as Surrogate Models for No-box Adversarial Attacks

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Jiaming Qiu, Lingyu Yi, Qi Li, Yige Sang, Jitao Xu, Changsheng Yeung, Dit-Yan Department of Computer Science and Engineering Hong Kong University of Science and Technology Hong Kong Department of Mathematics and Applications University of Naples Federico II Naples Italy School of Computing and Information Systems Singapore Management University Singapore School of Computer and Information Technology Beijing Key Laboratory of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China National Lab of Pattern Recognition Institute of Automation Chinese Academy of Sciences Beijing China University of Chinese Academy of Sciences Beijing100190 China

The vulnerability of Deep Neural Networks (DNNs) to adversarial attacks poses a significant challenge to their deployment in safety-critical applications. While extensive research has addressed various attack scenarios, the no-box attack setting—where adversaries have no prior knowledge, including access to training data of the target model—remains relatively underexplored despite its practical relevance. This work presents a systematic investigation into leveraging large-scale Vision-Language Models (VLMs), particularly CLIP, as surrogate models for executing no-box attacks. Our theoretical and empirical analyses reveal a key limitation in the execution of no-box attacks stemming from insufficient discriminative capabilities for direct application of vanilla CLIP as a surrogate model . To address this limitation, we propose MF-CLIP (Margin-based Fine-tuned CLIP), a novel framework that enhances CLIP’s effectiveness as a surrogate model through margin-aware feature space optimization. Comprehensive evaluations across diverse architectures and datasets demonstrate that MF-CLIP substantially advances the state-of-the-art in no-box attacks, surpassing existing baselines by 15.23% on standard models and achieving a 9.52% improvement on adversarially trained models. Our code (attached in the supplement) will be made publicly available to facilitate reproducibility and future research in this direction. Copyright © 2023, The Authors. All rights reserved.

关键词： Visual languages

Emotional Conversation Generation with Heterogeneous Graph Neural Network

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Liang, Yunlong Meng, Fandong Zhang, Ying Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

The successful emotional conversation system depends on sufficient perception and appropriate expression of emotions. In a real-life conversation, humans firstly instinctively perceive emotions from multi-source information, including the emotion flow hidden in dialogue history, facial expressions, audio, and personalities of speakers. Then, they convey suitable emotions according to their personalities, but these multiple types of information are insufficiently exploited in emotional conversation fields. To address this issue, in this paper, we propose a heterogeneous graph-based model for emotional conversation generation. Firstly, we design a Heterogeneous Graph-Based Encoder to represent the conversation content (i.e., the dialogue history, its emotion flow, facial expressions, audio, and speakers' personalities) with a heterogeneous graph neural network, and then predict suitable emotions for feedback. Secondly, we employ an Emotion-Personality-Aware Decoder to generate a response relevant to the conversation context as well as with appropriate emotions, through taking the encoded graph representations, the predicted emotions by the encoder and the personality of the current speaker as inputs. Experiments on both automatic and human evaluation show that our method can effectively perceive emotions from multi-source knowledge and generate a satisfactory response. Furthermore, based on the up-to-date text generator BART, our model still can achieve consistent improvement, which significantly outperforms some existing state-of-the-art models. Copyright © 2020, The Authors. All rights reserved.

关键词： Graph neural networks

An iterative multi-knowledge transfer network for aspect-based sentiment analysis

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Liang, Yunlong Meng, Fandong Zhang, Jinchao Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc. China

Aspect-based sentiment analysis (ABSA) mainly involves three subtasks: aspect term extraction, opinion term extraction, and aspect-level sentiment classification, which are typically handled in a separate or joint manner. However, previous approaches do not well exploit the interactive relations among three subtasks and do not pertinently leverage the easily available document-level labeled domain/sentiment knowledge, which restricts their performances. To address these issues, we propose a novel Iterative Multi-Knowledge Transfer Network (IMKTN) for end-to-end ABSA. For one thing, through the interactive correlations between the ABSA subtasks, our IMKTN transfers the task-specific knowledge from any two of the three subtasks to another one at the token level by utilizing a well-designed routing algorithm, that is, any two of the three subtasks will help the third one. For another, our IMKTN pertinently transfers the document-level knowledge, i.e., domain-specific and sentiment-related knowledge, to the aspect-level subtasks to further enhance the corresponding performance. Experimental results on three benchmark datasets demonstrate the effectiveness and superiority of our approach. Copyright © 2020, The Authors. All rights reserved.

关键词： Sentiment analysis