检索结果-内蒙古大学图书馆

BJTU-WeChat's Systems for the WMT22 Chat Translation Task 7

学校读者我要写书评

暂无评论

BJTU-WeChat's Systems for the WMT22 Chat Translation Task

7th Conference on Machine Translation, WMT 2022

作者： Liang, Yunlong Meng, Fandong Xu, Jinan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

ISBN: (纸本)9781959429296

This paper introduces the joint submission of the Beijing Jiaotong University and WeChat AI to the WMT'22 chat translation task for English↔German. Based on the Transformer (Vaswani et al., 2017), we apply several effective variants. In our experiments, we utilize the pre-training-then-fine-tuning paradigm. In the first pre-training stage, we employ data filtering and synthetic data generation (i.e., back-translation, forward-translation, and knowledge distillation). In the second fine-tuning stage, we investigate speaker-aware in-domain data generation, speaker adaptation, prompt-based context modeling, target denoising fine-tuning (Meng et al., 2020), and boosted self-COMET-based model ensemble. Our systems achieve 0.810 and 0.946 COMET (Rei et al., 2020) scores on English→German and German→English, respectively. The COMET scores of English→German and German→English are the highest among all submissions. © 2022 Association for Computational Linguistics.

关键词：

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

学校读者我要写书评

暂无评论

Cross-Align: Modeling Deep Cross-lingual Interactions for Wo...

2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022

作者： Lai, Siyu Yang, Zhen Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Word alignment which aims to extract lexicon translation equivalents between source and target sentences, serves as a fundamental tool for natural language processing. Recent studies in this area have yielded substantial improvements by generating alignments from contextualized embeddings of the pre-trained multilingual language models. However, we find that the existing approaches capture few interactions between the input sentence pairs, which degrades the word alignment quality severely, especially for the ambiguous words in the monolingual context. To remedy this problem, we propose Cross-Align to model deep interactions between the input sentence pairs, in which the source and target sentences are encoded separately with the shared self-attention modules in the shallow layers, while cross-lingual interactions are explicitly constructed by the cross-attention modules in the upper layers. Besides, to train our model effectively, we propose a two-stage training framework, where the model is trained with a simple Translation Language Modeling (TLM) objective in the first stage and then finetuned with a self-supervised alignment objective in the second stage. Experiments show that the proposed Cross-Align achieves the state-of-the-art (SOTA) performance on four out of five language pairs. © 2022 Association for Computational Linguistics.

关键词： Alignment

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Chen, Yijie Liu, Yijin Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel cross-tokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families (i.e, LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements. The code is available at https://***/pppa2019/ContexualDynamicMapping Copyright © 2025, The Authors. All rights reserved.

关键词： Teaching

D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wang, Panpan Niu, Liqiang Meng, Fandong Xu, Jinan Chen, Yufeng Zhou, Jie Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

In the domain of image generation, latent-based generative models occupy a dominant status;however, these models rely heavily on image tokenizer. To meet modeling requirements, autoregressive models possessing the characteristics of scalability and flexibility embrace a discrete-valued tokenizer, but face the challenge of poor image generation quality. In contrast, diffusion models take advantage of the continuous-valued tokenizer to achieve better generation quality but are subject to low efficiency and complexity. The existing hybrid models are mainly to compensate for information loss and simplify the diffusion learning process. The potential of merging discrete-valued and continuous-valued tokens in the field of image generation has not yet been explored. In this paper, we propose D2C, a novel two-stage method to enhance model generation capacity. In the first stage, the discrete-valued tokens representing coarse-grained image features are sampled by employing a small discrete-valued generator. Then in the second stage, the continuous-valued tokens representing fine-grained image features are learned conditioned on the discrete token sequence. In addition, we design two kinds of fusion modules for seamless interaction. On the ImageNet-256 benchmark, extensive experiment results validate that our model achieves superior performance compared with several continuous-valued and discrete-valued generative models on the class-conditional image generation tasks. © 2025, CC BY.

关键词： Coarse-grained modeling

Warmup-Distill: Bridge the Distribution Mismatch between Teacher and Student before Knowledge Distillation

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Sun, Zengkui Liu, Yijin Meng, Fandong Chen, Yufeng Xu, Jinan Zhou, Jie Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

The widespread deployment of Large Language Models (LLMs) is hindered by the high computational demands, making knowledge distillation (KD) crucial for developing compact smaller ones. However, the conventional KD methods endure the distribution mismatch issue between the teacher and student models, leading to the poor performance of distillation. For instance, the widely-used KL-based methods suffer the mode-averaging and mode-collapsing problems, since the mismatched probabitliy distribution between both models. Previous studies mainly optimize this issue via different distance calculations towards the distribution of both models. Unfortunately, the distribution mismatch issue still exists in the early stage of the distillation. Hence, to reduce the impact of distribution mismatch, we propose a simple yet efficient method, named WarmupDistill, which aligns the distillation of the student to that of the teacher in advance of distillation. Specifically, we first detect the distribution of the student model in practical scenarios with its internal knowledge, and then modify the knowledge with low probability via the teacher as the checker. Consequently, WarmupDistill aligns the internal student’s knowledge to that of the teacher, which expands the distribution of the student with the teacher’s, and assists the student model to learn better in the subsequent distillation. Experiments on the seven benchmarks demonstrate that WarmupDistill could provide a warmup student more suitable for distillation, which outperforms the vanilla student by as least +0.4 averaged score among all benchmarks. Noteably, with the assistance of Warmup-Distill, the distillation on the math task could yield a further improvement, at most +1.9% accuracy. Copyright © 2025, The Authors. All rights reserved.

关键词： Students

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

学校读者我要写书评

暂无评论

arXiv 2024年

关键词： Neurons

Beyond Binary Gender: Evaluating Gender-Inclusive Machine Translation with Ambiguous Attitude Words

学校读者我要写书评

暂无评论

arXiv 2024年

Gender bias has been a focal point in the study of bias in machine translation and language models. Existing machine translation gender bias evaluations are primarily focused on male and female genders, limiting the scope of the evaluation. To assess gender bias accurately, these studies often rely on calculating the accuracy of gender pronouns or the masculine and feminine attributes of grammatical gender via the stereotypes triggered by occupations or sentiment words (i.e., clear positive or negative attitude), which cannot extend to non-binary groups. This study presents a benchmark AmbGIMT (Gender-Inclusive Machine Translation with Ambiguous attitude words), which assesses gender bias beyond binary gender. Meanwhile, we propose a novel process to evaluate gender bias based on the Emotional Attitude Score (EAS), which is used to quantify ambiguous attitude words. In evaluating three recent and effective open-source LLMs and one powerful multilingual translation-specific model, our main observations are: (1) The translation performance within non-binary gender contexts is markedly inferior in terms of translation quality and exhibits more negative attitudes than binary-gender contexts. (2) The analysis experiments indicate that incorporating constraint context in prompts for gender identity terms can substantially reduce translation bias, while the bias remains evident despite the presence of the constraints. The code is publicly available at https://***/pppa2019/ambGIMT. Copyright © 2024, The Authors. All rights reserved.

关键词： Machine translation

A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Shen, Jiaxin Xu, Jinan Hu, Huiqi Lin, Luyi Zheng, Fei Ma, Guoyang Meng, Fandong Zhou, Jie Han, Wenjuan Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence Beijing China Law School Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored. We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation. Simultaneously, we propose an agent framework that employs a comprehensive suite of legal analysis tools to address the challenge task. This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the "Intelligent Court"1 © 2025, CC BY.

关键词： Benchmarking

Unified Model Learning for Various Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liang, Yunlong Meng, Fandong Xu, Jinan Wang, Jiaan Chen, Yufeng Zhou, Jie Beijing Key Lab of Traffic Data Analysis and Mining Beijing Jiaotong University Beijing China Pattern Recognition Center WeChat AI Tencent Inc China

Existing neural machine translation (NMT) studies mainly focus on developing dataset-specific models based on data from different tasks (e.g., document translation and chat translation). Although the dataset-specific models have achieved impressive performance, it is cumbersome as each dataset demands a model to be designed, trained, and stored. In this work, we aim to unify these translation tasks into a more general setting. Specifically, we propose a "versatile" model, i.e., the Unified Model Learning for NMT (UMLNMT) that works with data from different tasks, and can translate well in multiple settings simultaneously, and theoretically it can be as many as possible. Through unified learning, UMLNMT is able to jointly train across multiple tasks, implementing intelligent on-demand translation. On seven widely-used translation tasks, including sentence translation, document translation, and chat translation, our UMLNMT results in substantial improvements over dataset-specific models with significantly reduced model deployment costs. Furthermore, UMLNMT can achieve competitive or better performance than state-of-the-art dataset-specific methods. Human evaluation and in-depth analysis also demonstrate the superiority of our approach on generating diverse and high-quality translations. Additionally, we provide a new genre translation dataset about famous aphorisms with 186k Chinese→English sentence pairs. Copyright © 2023, The Authors. All rights reserved.

关键词： Neural machine translation