检索结果-内蒙古大学图书馆

Learning universal representations from word to sentence

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Li, Yian Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University China

Despite the well-developed cut-edge representation learning for language, most language representation models usually focus on specific level of linguistic unit, which cause great inconvenience when being confronted with handling multiple layers of linguistic objects in a unified way. Thus this work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space through a task-indep.ndent evaluation. We present our approach of constructing analogy datasets in terms of words, phrases and sentences and experiment with multiple representation models to examine geometric properties of the learned vector space. Then we empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation. Especially, our implementation of fine-tuning ALBERT on NLI and PPDB datasets achieves the highest accuracy on analogy tasks in different language levels. Further experiments on the insurance FAQ task show effectiveness of universal representation models in real-world applications. Copyright © 2020, The Authors. All rights reserved.

关键词： Linguistics

Multi-choice Dialogue-Based Reading Comprehension with Knowledge and key Turns

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Li, Junlong Zhang, Zhuosheng Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

Multi-choice machine reading comprehension (MRC) requires models to choose the correct answer from candidate options given a passage and a question. Our research focuses dialogue-based MRC, where the passages are multi-turn dialogues. It suffers from two challenges, the answer selection decision is made without support of latently helpful commonsense, and the multi-turn context may hide considerable irrelevant information. This work thus makes the first attempt to tackle those two challenges by extracting substantially important turns and utilizing external knowledge to enhance the representation of context. In this paper, the relevance of each turn to the question are calculated to choose key turns. Besides, terms related to the context and the question in a knowledge graph are extracted as external knowledge. The original context, question and external knowledge are encoded with the pre-trained language model, then the language representation and key turns are combined together with a will-designed mechanism to predict the answer. Experimental results on a DREAM dataset show that our proposed model achieves great improve- Copyright © 2020, The Authors. All rights reserved.

关键词： Knowledge graph

Semantics-Aware Inferential Network for Natural Language Understanding

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Zhang, Shuailiang Zhao, Hai Zhou, Junru Department of Computer Science and Engineering Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

For natural language understanding tasks, either machine reading comprehension or natural language inference, both semantics-aware and inference are favorable features of the concerned modeling for better understanding performance. Thus we propose a Semantics-Aware Inferential Network (SaiN) to meet such a motivation. Taking explicit contextualized semantics as a complementary input, the inferential module of SaiN enables a series of reasoning steps over semantic clues through an attention mechanism. By stringing these steps, the inferential network effectively learns to perform iterative reasoning which incorporates both explicit semantics and contextualized representations. In terms of well pre-trained language models as front-end encoder, our model achieves significant improvement on 11 tasks including machine reading comprehension and natural language inference. Copyright © 2020, The Authors. All rights reserved.

关键词： Semantics

Accurate word representations with universal visual guidance

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Zhang, Zhuosheng Yu, Haojie Zhao, Hai Wang, Rui Utiyama, Masao Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

Word representation is a fundamental component in neural language understanding models. Recently, pre-trained language models (PrLMs) offer a new performant method of contextualized word representations by leveraging the sequence-level context for modeling. Although the PrLMs generally give more accurate contextualized word representations than non-contextualized models do, they are still subject to a sequence of text contexts without diverse hints for word representation from multimodality. This paper thus proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. In detail, we build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. The texts and paired images are encoded in parallel, followed by an attention layer to integrate the multimodal representations. We show that the method substantially improves the accuracy of disambiguation. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach. Copyright © 2020, The Authors. All rights reserved.

关键词： Modeling languages

Document-level neural machine translation with document embeddings

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Jiang, Shu Zhao, Hai Li, Zuchao Lu, Bao-Liang Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

Standard neural machine translation (NMT) is on the assumption of document-level context indep.ndent. Most existing document-level NMT methods are satisfied with a smattering sense of brief document-level information, while this work focuses on exploiting detailed document-level context in terms of multiple forms of document embeddings, which is capable of sufficiently modeling deeper and richer document-level context. The proposed document-aware NMT is implemented to enhance the Transformer baseline by introducing both global and local document-level clues on the source end. Experiments show that the proposed method significantly improves the translation performance over strong baselines and other related studies. Copyright © 2020, The Authors. All rights reserved.

关键词： Neural machine translation

Enhancing Pre-trained Language Model with Lexical Simplification

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Bao, Rongzhou Wang, Jiayi Zhang, Zhuosheng Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

For both human readers and pre-trained language models (PrLMs), lexical diversity may lead to confusion and inaccuracy when understanding the underlying semantic meanings of given sentences. By substituting complex words with simple alternatives, lexical simplification (LS) is a recognized method to reduce such lexical diversity, and therefore to improve the understandability of sentences. In this paper, we leverage LS and propose a novel approach which can effectively improve the performance of PrLMs in text classification. A rule-based simplification process is applied to a given sentence. PrLMs are encouraged to predict the real label of the given sentence with auxiliary inputs from the simplified version. Using strong PrLMs (BERT and ELECTRA) as baselines, our approach can still further improve the performance in various text classification tasks. Copyright © 2020, The Authors. All rights reserved.

关键词： Semantics

Deep feature guided image retargeting 34

学校读者我要写书评

暂无评论

Deep feature guided image retargeting

34th IEEE International Conference on Visual Communications and Image Processing, VCIP 2019

作者： Wu, Jinan Xie, Rong Song, Li Liu, Bo Shanghai Jiao Tong University Institute of Image Communication and Network Engineering China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai200240 China University of Technology School of Computer Science Sydney Australia

ISBN: (纸本)9781728137230

Image retargeting is the technique to display images via devices with various aspect ratios and sizes. Traditional content-Aware retargeting methods rely on low-level features to predict pixel-wise importance and can hardly preserve both the structure lines and salient regions of the source image. To address this problem, we propose a novel adaptive image warping approach which integrates with deep convolutional neural network. In the proposed method, a visual importance map and a foreground mask map are generated by a pre-Trained network. The two maps and other constraints guide the warping process to yield retargeted results with less distortions. Extensive experiments in terms of visual quality and a user study are carried out on the widely used RetargetMe dataset. Experimental results show that our method outperforms current state-of-Art image retargeting methods. © 2019 IEEE.

关键词： Aspect ratio

Graph-to-sequence neural machine translation

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Duan, Sufeng Zhao, Hai Wang, Rui Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University China Kyoto Japan

Neural machine translation (NMT) usually works in a seq2seq learning way by viewing either source or target sentence as a linear sequence of words, which can be regarded as a special case of graph, taking words in the sequence as nodes and relationships between words as edges. In the light of the current NMT models more or less capture graph information among the sequence in a latent way, we present a graph-to-sequence model facilitating explicit graph information capturing. In detail, we propose a graph-based SAN-based NMT model called Graph-Transformer by capturing information of subgraphs of different orders in every layers. Subgraphs are put into different groups according to their orders, and every group of subgraphs respectively reflect different levels of dep.ndency between words. For fusing subgraph representations, we empirically explore three methods which weight different groups of subgraphs of different orders. Results of experiments on WMT14 English-German and IWSLT14 German-English show that our method can effectively boost the Transformer with an improvement of 1.1 BLEU points on WMT14 English-German dataset and 1.0 BLEU points on IWSLT14 German-English dataset. Copyright © 2020, The Authors. All rights reserved.

关键词： Neural machine translation

Capsule-Transformer for Neural Machine Translation

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Duan, Sufeng Cao, Juncheng Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

Transformer hugely benefits from its key design of the multi-head self-attention network (SAN), which extracts information from various perspectives through transforming the given input into different subspaces. However, its simple linear transformation aggregation strategy may still potentially fail to fully capture deeper contextualized information. In this paper, we thus propose the capsule-Transformer, which extends the linear transformation into a more general capsule routing algorithm by taking SAN as a special case of capsule network. So that the resulted capsule-Transformer is capable of obtaining a better attention distribution representation of the input sequence via information aggregation among different heads and words. Specifically, we see groups of attention weights in SAN as low layer capsules. By applying the iterative capsule routing algorithm they can be further aggregated into high layer capsules which contain deeper contextualized information. Experimental results on the widely-used machine translation datasets show our proposed capsule-Transformer outperforms strong Transformer baseline significantly. Copyright © 2020, The Authors. All rights reserved.

关键词： Iterative methods