检索结果-内蒙古大学图书馆

Momentum Decoding: Open-ended Text Generation As Graph Exploration

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Lan, Tian Su, Yixuan Liu, Shuhang Huang, Heyan Mao, Xian-Ling School of Computer Science and Technology Beijing Institute of Technology China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications China

Open-ended text generation with autoregressive language models (LMs) is one of the core tasks in natural language processing. However, maximization-based decoding methods (e.g., greedy/beam search) often lead to the degeneration problem, i.e., the generated text is unnatural and contains undesirable repetitions. Existing solutions to this problem either introduce randomness prone to incoherence or require a look-ahead mechanism that demands extra computational overhead. In this study, we formulate open-ended text generation from a new perspective, i.e., we view it as an exploration process within a directed graph. Thereby, we understand the phenomenon of degeneration as circular loops within the directed graph. Based on our formulation, we propose a novel decoding method—momentum decoding—which encourages the LM to greedily explore new nodes outside the current graph. Meanwhile, it also allows the LM to return to the existing nodes with a momentum downgraded by a pre-defined resistance function. We extensively test our approach on three benchmarks from different domains through automatic and human evaluations. The results show that momentum decoding performs comparably with the current state of the art while enjoying notably improved inference speed and computation FLOPs. Furthermore, we conduct a detailed analysis to reveal the merits and inner workings of our approach. © 2022, CC BY.

关键词： Momentum

Topical sentence embedding for query focused document summarization

学校读者我要写书评

暂无评论

Topical sentence embedding for query focused document summar...

2017 IJCAI Workshop on Semantic Machine Learning, SML 2017

作者： Gao, Yang Wei, Linjing Huang, Heyan Liu, Qian Beijing Eng. Res. Ctr. of High Vol. Language Information Processing and Cloud Computing Applications China BIT Beijing Advanced Innovation Center for Imaging Technology Capital Normal University China

Distributed vector representation for sentences have been utilized in summarization area, since it simplifies semantic cosine calculation between sentence to sentence as well as sentence to document. Many extension works have been done to incorporate latent topics and word embedding, however, few of them assign sentences with explicit topics. Besides, much sentence embedding framework follows the same spirit of prediction task about a word in the sentence, which omits the sentence-to-sentence coherence. To address these problems, we proposed a novel sentence embedding framework to collaborate the current sentence representation, word-based content and topic assignment of the sentence to predict the next sentence representation. The experiments on summarization tasks show our model outperforms state-of-the-art methods. Copyright © by the paper's authors. Copying permitted for private and academic purposes.

关键词： Semantics

Utilizing Crowdsourcing for the Construction of Chinese-Mongolian Speech Corpus with Evaluation Mechanism

学校读者我要写书评

暂无评论

Utilizing Crowdsourcing for the Construction of Chinese-Mong...

The Third International Conference of Pioneering Computer Scientists, Engineers and Educators,ICPCSEE 2017(originally ICYCSEE)

作者： Rihai Su Shumin Shi Meng Zhao Heyan Huang School of Computer Sciences and Technology Beijing Institute of Technology Beijing Engineering Research Centre of High Volume Language Information Processing and Cloud Computing Applications

Crowdsourcing has been used recently as an alternative to traditional costly annotation by many natural language processing groups. In this paper, we explore the use of Wechat Official Account Platform(WOAP) in order to build a speech corpus and to assess the feasibility of using WOAP followers(also known as contributors) to assemble speech corpus of Mongolian. A Mongolian language qualification test was used to filter out potential non-qualified participants. We gathered natural speech recordings in our daily life, and constructed a ChineseMongolian Speech Corpus(CMSC) of 31472 utterances from 296 native speakers who are fluent in Mongolian, totalling 30.8 h of speech. Then,an evaluation experiment was performed, in where the contributors were asked to choose a correct sentence from a multiple choice list to ensure the high-quality of corpus. The results obtained so far showed that crowdsourcing for constructing CMSC with an evaluation mechanism could be more effective than traditional experiments requiring expertise.

关键词： Crowdsourcing Chinese-Mongolian corpus Speech corpus WOAP Evaluation mechanism

Neural Chinese word segmentation as sequence to sequence translation

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Shi, Xuewen Huang, Heyan Jian, Ping Guo, Yuhang Wei, Xiaochi Tang, Yikun Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications School of Computer Science and Technology Beijing Institute of Technology Beijing100081 China

Recently, Chinese word segmentation (CWS) methods using neural networks have made impressive progress. Most of them regard the CWS as a sequence labeling problem which construct models based on local features rather than considering global information of input sequence. In this paper, we cast the CWS as a sequence translation problem and propose a novel sequence-to-sequence CWS model with an attention-based encoder-decoder framework. The model captures the global information from the input and directly outputs the segmented sequence. It can also tackle other NLP tasks with CWS jointly in an end-to-end mode. Experiments on Weibo, PKU and MSRA benchmark datasets show that our approach has achieved competitive performances compared with state-of-the-art methods. Meanwhile, we successfully applied our proposed model to jointly learning CWS and Chinese spelling correction, which demonstrates its applicability of multi-task fusion. Copyright © 2019, The Authors. All rights reserved.

关键词： Natural language processing systems

Concept pointer network for abstractive summarization

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Wenbo, Wang Yang, Gao Heyan, Huang Yuxiang, Zhou School of Computer Science and Technology Beijing Institute of Technology Beijing100081 China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications

A quality abstractive summary should not only copy salient source texts as summaries but should also tend to generate new conceptual words to express concrete details. Inspired by the popular pointer generator sequence-to-sequence model, this paper presents a concept pointer network for improving these aspects of abstractive summarization. The network leverages knowledge-based, context-aware conceptualizations to derive an extended set of candidate concepts. The model then points to the most appropriate choice using both the concept set and original source text. This joint approach generates abstractive summaries with higher-level semantic concepts. The training model is also optimized in a way that adapts to different data, which is based on a novel method of distantly-supervised learning guided by reference summaries and testing set. Overall, the proposed approach provides statistically significant improvements over several state-of-the-art models on both the DUC-2004 and Gigaword datasets. A human evaluation of the model's abstractive abilities also supports the quality of the summaries produced within this *** Codes 68U15, 68T50 Copyright © 2019, The Authors. All rights reserved.

关键词： Knowledge based systems

Tag recommendation by word-level tag sequence modeling

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Shi, Xuewen Huang, Heyan Zhao, Shuyang Jian, Ping Tang, Yi-Kun School of Computer Science and Technology Beijing Institute of Technology Beijing100081 China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications

Recently, neural network based tag recommendation has gained much attention. Most of these methods regard tag recommendation as multi-label classification. They treat different tags as individual categories with fixed labels, without considering the rich relations among tags. Moreover, using a fixed number of categories cannot tackle dynamic, changing tags with ongoing topics. In this paper, we transform tag recommendation into a word-based text generation problem and introduce a sequence-to-sequence model. For efficiently modeling the semantic dependencies among tags in tag sequence and the strong sequential relations among the tag-words, we propose an essential sequence-to-sequence model, named LSTM-Attention. The model inherits the advantages of recurrent network based encoder for sequential modeling and attention based decoder for learning relations globally. In addition, as a text generation method, the proposed model is able to generate unseen tags, which is more applicable and flexible to real scenarios. Extensive experimental results on two datasets, i.e., Zhihu and Weibo, clearly illustrate the proposed model significantly outperforms other state-of-the-art text classification based methods and well demonstrate its advantage of handling unseen tags. Copyright © 2019, The Authors. All rights reserved.

关键词： Text processing

A hierarchy-based constraint dependency grammar parsing for Chinese

学校读者我要写书评

暂无评论

A hierarchy-based constraint dependency grammar parsing for ...

International Conference on Audio, language and Image processing, ICALIP

作者： Peng Li Lejian Liao Xin Li Beijing Engineering Research Centre of High Volume Language Information Processing & Cloud Computing Applications School of Computer Science Beijing Institute of Technology Beijing China

Constraint Dependency Grammar (CDG) is a well known formalism which represents the grammatical rules as constraints on dependency relations. It reduces the structural ambiguities via constraint-propagation instead of generating individual parse trees as the conventional works. In our previous work, we have successfully adopted CDG in Chinese [5]. In this paper, we tend to further explore the nature of the constraints for Chinese CDG. We propose to develop a schema of three hierarchies for the constraints, namely in-the-phrase, between-the-phrase and between-the-simple-sentence, which is based on a study of constraints in corpus with a prior knowledge. With this proposed schema, we could further diminish the ambiguities caused by the originally defined constraints which is lack of hierarchical grammar information considered, The experiments are conducted and the results support our claims.

关键词： Grammar Syntactics Filtering information processing Educational institutions Semantics Presses

QACtools: A Quality Assessment and Quality Control Tool for Next-Generation Sequencing Data

学校读者我要写书评

暂无评论

QACtools: A Quality Assessment and Quality Control Tool for ...

2015年中国智能自动化学术会议

作者： Dandan Song Ning Li Lejian Liao Beijing Lab of Intelligent Information Technology School of Computer Science and Technology Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications Beijing Institute of Technology

In recent years, next-generation sequencing technologies have been widely used, and are rapidly changing the landscape of genetics with unimaginable speed. However, the preliminary quality control regulations including trimming adaptor sequences, quality statistics, and depth and coverage statistics need to be applied to raw reads before further analysis. QACtools implements various utilities for processing raw reads in the FASTQ format and alignments in the BAM *** analyzes some relevant properties of next-generation sequencing data such as average read length, base quality scores, and so on. Additionally, to our knowledge,none of the other tools support depth and coverage statistic that have been provided in our tool for analysis of large DNA sequencing data. It is crucial for overcoming errors in base calling and assembly. We suppose tool is useful for the quality control of NGS data to perform further analysis. This software is an open source application freely available at https://***/projects/qactools/.

关键词： NGS data analysis Quality control FASTQ format BAM format

Dynamic Regret with Unknown Delays

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Wu, Ping Liu, Zhengyang Lu, Haolin Huang, Heyan School of Computer Science and Technology Beijing Institute of Technology Beijing100081 China Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications China

We study the problem of online convex optimization in dynamic environments with unknown and arbitrary delays. Without delays, existing work has achieved the optimal $\Theta(\sqrt{(1+\Delta_T)T})$ regret bound for any given sequence of comparators, where $T$ is the time horizon and $\Delta_T$ is the path-length of the comparator sequence. However, their algorithm assumes that the loss function can be revealed immediately to the agent after making the decision. In this paper, we propose a novel algorithm to tackle the delayed setting. We show that it enjoys $\mathcal{O}(\max\{\sqrt{(1+\Delta_T)D_T},(\hat{d}-1)\Delta_T\})$ regret bound, where $\hat{d}$ is the maximum delay and $D_T$ is the cumulative delay over the whole horizon. To complement our algorithm, we also establish an $\Omega(\sqrt{(1+\Delta_T)D_T})$ lower bound. The basic idea is to employ a delayed meta algorithm to track parallel delayed expert algorithms with varying step sizes. We further extend our algorithm to the case that we have no idea about the time horizon and cumulative delay using a novel doubling trick. Finally, numerical simulations are provided to illustrate the validity of our theoretical results. © 2023, The Authors. All rights reserved.

关键词： Comparators (optical)