检索结果-内蒙古大学图书馆

BJTU-WeChat's Systems for the WMT22 Chat Translation Task 7

学校读者我要写书评

暂无评论

BJTU-WeChat's Systems for the WMT22 Chat Translation Task

7th Conference on Machine Translation, WMT 2022

作者： Liang, Yunlong Meng, Fandong Xu, Jinan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

ISBN: (纸本)9781959429296

This paper introduces the joint submission of the Beijing Jiaotong University and WeChat AI to the WMT'22 chat translation task for English↔German. Based on the Transformer (Vaswani et al., 2017), we apply several effective variants. In our experiments, we utilize the pre-training-then-fine-tuning paradigm. In the first pre-training stage, we employ data filtering and synthetic data generation (i.e., back-translation, forward-translation, and knowledge distillation). In the second fine-tuning stage, we investigate speaker-aware in-domain data generation, speaker adaptation, prompt-based context modeling, target denoising fine-tuning (Meng et al., 2020), and boosted self-COMET-based model ensemble. Our systems achieve 0.810 and 0.946 COMET (Rei et al., 2020) scores on English→German and German→English, respectively. The COMET scores of English→German and German→English are the highest among all submissions. © 2022 Association for Computational Linguistics.

关键词：

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

学校读者我要写书评

暂无评论

Cross-Align: Modeling Deep Cross-lingual Interactions for Wo...

2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

作者： Lai, Siyu Yang, Zhen Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models. However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the cross-attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs. © 2022 Association for Computational Linguistics.

关键词： Alignment

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Chen, Yijie Liu, Yijin Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel cross-tokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families (i.e, LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements. The code is available at https://***/pppa2019/ContexualDynamicMapping Copyright © 2025, The Authors. All rights reserved.

关键词： Teaching

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

学校读者我要写书评

暂无评论

arXiv 2024年

关键词： Neurons

Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words

学校读者我要写书评

暂无评论

arXiv 2024年

Gender bias has been a focal point in the study of bias in machine translation and language models. Existing machine translation gender bias evaluations are primarily focused on male and female genders, limiting the scope of the evaluation. To assess gender bias accurately, these studies often rely on calculating the accuracy of gender pronouns or the masculine and feminine attributes of grammatical gender via the stereotypes triggered by occupations or sentiment words (i.e., clear positive or negative attitude), which cannot extend to non-binary groups. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words), which assesses gender bias beyond binary gender. Meanwhile, we propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words. In evaluating three recent and effective open-source LLMs and one powerful multilingual translation-specific model, our main observations are: (1) The translation performance within non-binary gender contexts is markedly inferior in terms of translation quality and exhibits more negative attitudes than binary-gender contexts. (2) The analysis experiments indicate that incorporating constraint context in prompts for gender identity terms can substantially reduce translation bias, while the bias remains evident despite the presence of the constraints. The code is publicly available at https://***/pppa2019/ambGIMT. Copyright © 2024, The Authors. All rights reserved.

关键词： Machine translation

Unified Model Learning for Various Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liang, Yunlong Meng, Fandong Xu, Jinan Wang, Jiaan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Existing neural machine translation (NMT) studies mainly focus on developing dataset-specific models based on data from different tasks (e.g., document translation and chat translation). Although the dataset-specific models have achieved impressive performance, it is cumbersome as each dataset demands a model to be designed, trained, and stored. In this work, we aim to unify these translation tasks into a more general setting. Specifically, we propose a "versatile" model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks, and can translate well in multiple settings simultaneously, and theoretically it can be as many as possible. Through unified learning, UMLNMT is able to jointly train across multiple tasks, implementing intelligent on-demand translation. On seven widely-used translation tasks, including sentence translation, document translation, and chat translation, our UMLNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs. Furthermore, UMLNMT can achieve competitive or better performance than state-of-the-art dataset-specific methods. Human evaluation and in-depth analysis also demonstrate the superiority of our approach on generating diverse and high-quality translations. Additionally, we provide a new genre translation dataset about famous aphorisms with 186k Chinese→English sentence pairs. Copyright © 2023, The Authors. All rights reserved.

关键词： Neural machine translation

Modeling bilingual conversational characteristics for neural chat translation 59

学校读者我要写书评

暂无评论

Modeling bilingual conversational characteristics for neural...

Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL-IJCNLP 2021

作者： Liang, Yunlong Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

ISBN: (纸本)9781954085527

Neural chat translation aims to translate bilingual conversational text, which has a broad application in international exchanges and cooperation. Despite the impressive performance of sentence-level and context-aware Neural Machine Translation (NMT), there still remain challenges to translate bilingual conversational text due to its inherent characteristics such as role preference, dialogue coherence, and translation consistency. In this paper, we aim to promote the translation quality of conversational text by modeling the above properties. Specifically, we design three latent variational modules to learn the distributions of bilingual conversational characteristics. Through sampling from these learned distributions, the latent variables, tailored for role preference, dialogue coherence, and translation consistency, are incorporated into the NMT model for better translation. We evaluate our approach on the benchmark dataset BConTrasT (English-German) and a self-collected bilingual dialogue corpus, named BMELD (English-Chinese). Extensive experiments show that our approach notably boosts the performance over strong baselines by a large margin and significantly surpasses some state-of-the-art context-aware NMT models in terms of BLEU and TER. Additionally, we make the BMELD dataset publicly available for the research community. © 2021 Association for Computational Linguistics

关键词： Neural machine translation

Generating Authentic Adversarial Examples beyond Meaning-preserving with Doubly Round-trip Translation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Lai, Siyu Yang, Zhen Meng, Fandong Zhang, Xue Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China Pattern Recognition Center WeChat AI Tencent Inc China

Generating adversarial examples for Neural Machine Translation (NMT) with single Round-Trip Translation (RTT) has achieved promising results by releasing the meaning-preserving restriction. However, a potential pitfall for this approach is that we cannot decide whether the generated examples are adversarial to the target NMT model or the auxiliary backward one, as the reconstruction error through the RTT can be related to either. To remedy this problem, we propose a new criterion for NMT adversarial examples based on the Doubly Round-Trip Translation (DRTT). Specifically, apart from the source-target-source RTT, we also consider the target-source-target one, which is utilized to pick out the authentic adversarial examples for the target NMT model. Additionally, to enhance the robustness of the NMT model, we introduce the masked language models to construct bilingual adversarial pairs based on DRTT, which are used to train the NMT model directly. Extensive experiments on both the clean and noisy test sets (including the artificial and natural noise) show that our approach substantially improves the robustness of NMT models. Copyright © 2022, The Authors. All rights reserved.

关键词： Neural machine translation

RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhou, Chulun Liang, Yunlong Meng, Fandong Xu, Jinan Su, Jinsong Zhou, Jie Pattern Recognition Center WeChat AI Tencent Inc. China Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China School of Informatics Xiamen University Xiamen China

Multilingual vision-language (V&L) pretraining has achieved remarkable progress in learning universal representations across different modalities and languages. In spite of recent success, there still remain challenges limiting further improvements of V&L pre-trained models in multilingual settings. Particularly, current V&L pre-training methods rely heavily on strictly-aligned multilingual image-text pairs generated from English-centric datasets through machine translation. However, the cost of collecting and translating such strictly-aligned datasets is usually unbearable. In this paper, we propose Regularized Contrastive Cross-lingual Cross-modal (RC3) pre-training, which further exploits more abundant weakly-aligned multilingual image-text pairs. Specifically, we design a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs according to textual relevance. Besides, existing V&L pre-training approaches mainly deal with visual inputs by either region-of-interest (ROI) features or patch embeddings. We flexibly integrate the two forms of visual features into our model for pre-training and downstream multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method over competitive contrast models with stronger zero-shot capability. © 2023, CC BY-NC-ND.

关键词： Image segmentation