检索结果-内蒙古大学图书馆

Successfully Applying the Stabilized Lottery Ticket Hypothesis to the Transformer Architecture

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Brix, Christopher Bahar, Parnia Ney, Hermann Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University AachenD-52056 Germany

Sparse models require less memory for storage and enable a faster inference by reducing the necessary number of FLOPs. This is relevant both for time-critical and on-device computations using neural networks. The stabilized lottery ticket hypothesis states that networks can be pruned after none or few training iterations, using a mask computed based on the unpruned converged model. On the transformer architecture and the WMT 2014 English→German and English→French tasks, we show that stabilized lottery ticket pruning performs similar to magnitude pruning for sparsity levels of up to 85%, and propose a new combination of pruning techniques that outperforms all other techniques for even higher levels of sparsity. Furthermore, we confirm that the parameter’s initial sign and not its specific value is the primary factor for successful training, and show that magnitude pruning cannot be used to find winning lottery tickets. Copyright © 2020, The Authors. All rights reserved.

关键词： Network architecture

RETURNN as a generic flexible neural toolkit with application to translation and speech recognition

学校读者我要写书评

暂无评论

arXiv 2018年

作者： Zeyer, Albert Alkhouli, Tamer Ney, Hermann Human Language Technology and Pattern Recognition Group RWTH Aachen University Aachen Germany AppTek United States NNAISENSE Switzerland

We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure TensorFlow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We obtain state-of-the-art models trained on the WMT 2017 German↔English translation task. We also present end-to-end model results for speech recognition on the Switchboard task. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications. Copyright © 2018, The Authors. All rights reserved.

关键词： Speech recognition

The RWTH Aachen Machine Translation System for WMT 2012 12

学校读者我要写书评

暂无评论

The RWTH Aachen Machine Translation System for WMT 2012

Workshop on Statistical Machine Translation

作者： Matthias Huck Stephan Peitz Markus Freitag Malte Nuhn Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen Germany

ISBN: (纸本)9781622765928

This paper describes the statistical machine translation (SMT) systems developed at RWTH Aachen University for the translation task of the NAACL 2012 Seventh Workshop on Statistical Machine Translation (WMT 2012). We participated in the evaluation campaign for the French-English and German-English language pairs in both translation directions. Both hierarchical and phrase-based SMT systems are applied. A number of different techniques are evaluated, including an insertion model, different lexical smoothing methods, a discriminative reordering extension for the hierarchical system, reverse translation, and system combination. By application of these methods we achieve considerable improvements over the respective baseline systems.

关键词： machine translation system machine translation Surface mount technology Hierarchical application methods Translations Translation Translation Process smoothing methods Hierarchical systems

Local system voting feature for machine translation system combination

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Freitag, Markus Peter, Jan-Thorsten Peitz, Stephan Feng, Minwei Ney, Hermann Human Language Technology Pattern Recognition Group Computer Science Department RWTH Aachen University AachenD-52056 Germany

In this paper, we enhance the traditional confusion network system combination approach with an additional model trained by a neural network. This work is motivated by the fact that the commonly used binary system voting models only assign each input system a global weight which is responsible for the global impact of each input system on all translations. This prevents individual systems with low system weights from having influence on the system combination output, although in some situations this could be helpful. Further, words which have only been seen by one or few systems rarely have a chance of being present in the combined output. We train a local system voting model by a neural network which is based on the words themselves and the combinatorial occurrences of the different system outputs. This gives system combination the option to prefer other systems at different word positions even for the same sentence. Copyright © 2017, The Authors. All rights reserved.

关键词： Machine learning

On Search Strategies for Document-Level Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Herold, Christian Ney, Hermann Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University AachenD-52056 Germany

Compared to sentence-level systems, document-level neural machine translation (NMT) models produce a more consistent output across a document and are able to better resolve ambiguities within the input. There are many works on document-level NMT, mostly focusing on modifying the model architecture or training strategy to better accommodate the additional context-input. On the other hand, in most works, the question on how to perform search with the trained model is scarcely discussed, sometimes not mentioned at all. In this work, we aim to answer the question how to best utilize a context-aware translation model in decoding. We start with the most popular document-level NMT approach and compare different decoding schemes, some from the literature and others proposed by us. In the comparison, we are using both, standard automatic metrics, as well as specific linguistic phenomena on three standard document-level translation benchmarks. We find that most commonly used decoding strategies perform similar to each other and that higher quality context information has the potential to further improve the translation. Copyright © 2023, The Authors. All rights reserved.

关键词： Neural machine translation

uniblock: Scoring and Filtering Corpus with Unicode Block Information

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Gao, Yingbo Wang, Weiyue Ney, Hermann Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University AachenD-52056 Germany

The preprocessing pipelines in Natural language Processing usually involve a step of removing sentences consisted of illegal characters. The definition of illegal characters and the specific removal strategy depend on the task, language, domain, etc, which often lead to tiresome and repetitive scripting of rules. In this paper, we introduce a simple statistical method, uniblock1, to overcome this problem. For each sentence, uniblock generates a fixed-size feature vector using Unicode block information of the characters. A Gaussian mixture model is then estimated on some clean corpus using variational inference. The learned model can then be used to score sentences and filter corpus. We present experimental results on Sentiment Analysis, language Modeling and Machine Translation, and show the simplicity and effectiveness of our method. Copyright © 2019, The Authors. All rights reserved.

关键词： Gaussian distribution

Online Learning for Neural Machine Translation Post-editing

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Peris, Álvaro Cebrián, Luis Casacuberta, Francisco Pattern Recognition and Human Language Technology Research Center Universitat Politècnica de València València Spain

Neural machine translation has meant a revolution of the field. Nevertheless, post-editing the outputs of the system is mandatory for tasks requiring high translation quality. Post-editing offers a unique opportunity for improving neural machine translation systems, using online learning techniques and treating the post-edited translations as new, fresh training data. We review classical learning methods and propose a new optimization algorithm. We thoroughly compare online learning algorithms in a post-editing scenario. Results show significant improvements in translation quality and effort reduction. Copyright © 2017, The Authors. All rights reserved.

关键词： Learning algorithms

Document-Level language Models for Machine Translation

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Petrick, Frithjof Herold, Christian Petrushkov, Pavel Khadivi, Shahram Ney, Hermann eBay Inc. Aachen Germany Human Language Technology and Pattern Recognition Group RWTH Aachen University Aachen Germany

Despite the known limitations, most machine translation systems today still operate on the sentence-level. One reason for this is, that most parallel training data is only sentence-level aligned, without document-level meta information available. In this work, we set out to build context-aware translation systems utilizing document-level monolingual data instead. This can be achieved by combining any existing sentence-level translation model with a document-level language model. We improve existing approaches by leveraging recent advancements in model combination. Additionally, we propose novel weighting techniques that make the system combination more flexible and significantly reduce computational overhead. In a comprehensive evaluation on four diverse translation tasks, we show that our extensions improve document-targeted scores substantially and are also computationally more efficient. However, we also find that in most scenarios, back-translation gives even better results, at the cost of having to re-train the translation system. Finally, we explore language model fusion in the light of recent advancements in large language models. Our findings suggest that there might be strong potential in utilizing large language models via model combination. Copyright © 2023, The Authors. All rights reserved.

关键词： Machine translation

Active learning for interactive neural machine translation of data streams

学校读者我要写书评

暂无评论

arXiv 2018年

作者： Peris, Álvaro Casacuberta, Francisco Pattern Recognition and Human Language Technology Research Center Universitat Politècnica de València València Spain

We study the application of active learning techniques to the translation of unbounded data streams via interactive neural machine translation. The main idea is to select, from an unbounded stream of source sentences, those worth to be supervised by a human agent. The user will interactively translate those samples. Once validated, these data is useful for adapting the neural machine translation model. We propose two novel methods for selecting the samples to be validated. We exploit the information from the attention mechanism of a neural machine translation system. Our experiments show that the inclusion of active learning techniques into this pipeline allows to reduce the effort required during the process, while increasing the quality of the translation system. Moreover, it enables to balance the human effort required for achieving a certain translation quality. Moreover, our neural system outperforms classical approaches by a large margin. Copyright © 2018, The Authors. All rights reserved.

关键词： Neural machine translation