检索结果-内蒙古大学图书馆

Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Lai, Siyu Yang, Zhen Meng, Fandong Zhang, Xue Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China Pattern Recognition Center WeChat AI Tencent Inc China

Generating adversarial examples for Neural Machine Translation (NMT) with single Round-Trip Translation (RTT) has achieved promising results by releasing the meaning-preserving restriction. However, a potential pitfall for this approach is that we cannot decide whether the generated examples are adversarial to the target NMT model or the auxiliary backward one, as the reconstruction error through the RTT can be related to either. To remedy this problem, we propose a new criterion for NMT adversarial examples based on the Doubly Round-Trip Translation (DRTT). Specifically, apart from the source-target-source RTT, we also consider the target-source-target one, which is utilized to pick out the authentic adversarial examples for the target NMT model. Additionally, to enhance the robustness of the NMT model, we introduce the masked language models to construct bilingual adversarial pairs based on DRTT, which are used to train the NMT model directly. Extensive experiments on both the clean and noisy test sets (including the artificial and natural noise) show that our approach substantially improves the robustness of NMT models. Copyright © 2022, The Authors. All rights reserved.

关键词： Neural machine translation

RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhou, Chulun Liang, Yunlong Meng, Fandong Xu, Jinan Su, Jinsong Zhou, Jie Pattern Recognition Center WeChat AI Tencent Inc. China Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China School of Informatics Xiamen University Xiamen China

Multilingual vision-language (V&L) pretraining has achieved remarkable progress in learning universal representations across different modalities and languages. In spite of recent success, there still remain challenges limiting further improvements of V&L pre-trained models in multilingual settings. Particularly, current V&L pre-training methods rely heavily on strictly-aligned multilingual image-text pairs generated from English-centric datasets through machine translation. However, the cost of collecting and translating such strictly-aligned datasets is usually unbearable. In this paper, we propose Regularized Contrastive Cross-lingual Cross-modal (RC3) pre-training, which further exploits more abundant weakly-aligned multilingual image-text pairs. Specifically, we design a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs according to textual relevance. Besides, existing V&L pre-training approaches mainly deal with visual inputs by either region-of-interest (ROI) features or patch embeddings. We flexibly integrate the two forms of visual features into our model for pre-training and downstream multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method over competitive contrast models with stronger zero-shot capability. © 2023, CC BY-NC-ND.

关键词： Image segmentation

Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhang, Songming Liu, Yijin Meng, Fandong Chen, Yufeng Xu, Jinan Liu, Jian Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Token-level adaptive training approaches can alleviate the token imbalance problem and thus improve neural machine translation, through re-weighting the losses of different target tokens based on specific statistical metrics (e.g., token frequency or mutual information). Given that standard translation models make predictions on the condition of previous target contexts, we argue that the above statistical metrics ignore target context information and may assign inappropriate weights to target tokens. While one possible solution is to directly take target contexts into these statistical metrics, the target-context-aware statistical computing is extremely expensive, and the corresponding storage overhead is unrealistic. To solve the above issues, we propose a target-context-aware metric, named conditional bilingual mutual information (CBMI), which makes it feasible to supplement target context information for statistical metrics. Particularly, our CBMI can be formalized as the log quotient of the translation model probability and language model probability by decomposing the conditional joint distribution. Thus CBMI can be efficiently calculated during model training without any pre-specific statistical calculations and large storage overhead. Furthermore, we propose an effective adaptive training approach based on both the token- and sentence-level CBMI. Experimental results on WMT14 English-German and WMT19 Chinese-English tasks show our approach can significantly outperform the Transformer baseline and other related methods. Copyright © 2022, The Authors. All rights reserved.

关键词： Statistics

Scheduled Multi-task Learning for Neural Chat Translation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Liang, Yunlong Meng, Fandong Xu, Jinan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

The goal of the Neural Chat Translation (NCT) is to translate conversational text into different languages. Existing methods mainly focus on modeling the bilingual dialogue characteristics (e.g., coherence) to improve chat translation via multi-task learning on small-scale chat translation data. Although the NCT models have achieved impressive success, it is still far from satisfactory due to insufficient chat translation data and simple joint training manners. To address the above issues, we propose a scheduled multi-task learning framework for NCT. Specifically, we devise a three-stage training framework to incorporate the large-scale in-domain chat translation data into training by adding a second pre-training stage between the original pre-training and fine-tuning stages. Further, we investigate where and how to schedule the dialogue-related auxiliary tasks in multiple training stages to effectively enhance the main chat translation task. Extensive experiments on four language directions (English↔Chinese and English↔German) demonstrate the effectiveness of the proposed approach. Additionally, we have made the large-scale in-domain paired bilingual dialogue dataset publicly available for the research community. Copyright © 2022, The Authors. All rights reserved.

关键词： Learning systems

BJTU-WeChat's Systems for the WMT22 Chat Translation Task

学校读者我要写书评

暂无评论

arXiv 2022年

This paper introduces the joint submission of the Beijing Jiaotong University and WeChat AI to the WMT’22 chat translation task for English↔German. Based on the Transformer (Vaswani et al., 2017), we apply several effective variants. In our experiments, we utilize the pre-training-thenfine-tuning paradigm. In the first pre-training stage, we employ data filtering and synthetic data generation (i.e., back-translation, forward-translation, and knowledge distillation). In the second fine-tuning stage, we investigate speaker-aware in-domain data generation, speaker adaptation, prompt-based context modeling, target denoising fine-tuning (Meng et al., 2020), and boosted self-COMET-based model ensemble. Our systems achieve 0.810 and 0.946 COMET (Rei et al., 2020) scores1 on English→German and German→English, respectively. The COMET scores of English→German and German→English are the highest among all submissions. Copyright © 2022, The Authors. All rights reserved.

关键词： Distillation

D2TV: Dual Knowledge Distillation and Target-oriented Vision Modeling for Many-to-Many Multimodal Summarization

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liang, Yunlong Meng, Fandong Wang, Jiaan Xu, Jinan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China School of Computer Science and Technology Soochow University Suzhou China

Many-to-many multimodal summarization (M3S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence, which essentially comprises multimodal monolingual summarization (MMS) and multimodal cross-lingual summarization (MXLS) tasks. Although much work has been devoted to either MMS or MXLS and has obtained increasing attention in recent years, little research pays attention to the M3S task. Besides, existing studies mainly focus on 1) utilizing MMS to enhance MXLS via knowledge distillation without considering the performance of MMS or 2) improving MMS models by filtering summary-unrelated visual features with implicit learning or explicitly complex training objectives. In this paper, we first introduce a general and practical task, i.e., M3S. Further, we propose a dual knowledge distillation and target-oriented vision modeling framework for the M3S task. Specifically, the dual knowledge distillation method guarantees that the knowledge of MMS and MXLS can be transferred to each other and thus mutually prompt both of them. To offer target-oriented visual features, a simple yet effective target-oriented contrastive objective is designed and responsible for discarding needless visual information. Extensive experiments on the many-to-many setting show the effectiveness of the proposed approach. Additionally, we will contribute a many-to-many multimodal summarization (M3Sum) dataset. Copyright © 2023, The Authors. All rights reserved.

关键词： Distillation

Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Liang, Yunlong Meng, Fandong Xu, Jinan Wang, Jiaan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Multimodal abstractive summarization (MAS) aims to produce a concise summary given the multimodal data (text and vision). Existing studies mainly focus on how to effectively use the visual features from the perspective of an article, having achieved impressive success on the high-resource English dataset. However, less attention has been paid to the visual features from the perspective of the summary, which may limit the model performance, especially in the low- and zero-resource scenarios. In this paper, we propose to improve the summary quality through summary-oriented visual features. To this end, we devise two auxiliary tasks including vision to summary task and masked image modeling task. Together with the main summarization task, we optimize the MAS model via the training objectives of all these tasks. By these means, the MAS model can be enhanced by capturing the summary-oriented visual features, thereby yielding more accurate summaries. Experiments on 44 languages, covering mid-high-, low-, and zero-resource scenarios, verify the effectiveness and superiority of the proposed approach, which achieves state-of-the-art performance under all scenarios. Additionally, we will contribute a large-scale multilingual multimodal abstractive summarization (MM-Sum) dataset. Copyright © 2022, The Authors. All rights reserved.

关键词： Large dataset

MSCTD: A Multimodal Sentiment Chat Translation dataset

学校读者我要写书评

暂无评论

arXiv 2022年

Multimodal machine translation and textual chat translation have received considerable attention in recent years. Although the conversation in its natural form is usually multimodal, there still lacks work on multimodal machine translation in conversations. In this work, we introduce a new task named Multimodal Chat Translation (MCT), aiming to generate more accurate translations with the help of the associated dialogue history and visual context. To this end, we firstly construct a Multimodal Sentiment Chat Translation dataset (MSCTD) containing 142,871 English-Chinese utterance pairs in 14,762 bilingual dialogues and 30,370 English-German utterance pairs in 3,079 bilingual dialogues. Each utterance pair, corresponding to the visual context that reflects the current conversational scene, is annotated with a sentiment label. Then, we benchmark the task by establishing multiple baseline systems that incorporate multimodal and sentiment features for MCT. Preliminary experiments on four language directions (English↔Chinese and English↔German) verify the potential of contextual and multimodal information fusion and the positive impact of sentiment on the MCT task. Additionally, as a by-product of the MSCTD, it also provides two new benchmarks on multimodal dialogue sentiment analysis. Our work can facilitate research on both multimodal chat translation and multimodal dialogue sentiment analysis. Copyright © 2022, The Authors. All rights reserved.

关键词： Sentiment analysis

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Lai, Siyu Yang, Zhen Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models. However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the cross-attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs.1 Copyright © 2022, The Authors. All rights reserved.

关键词： Alignment