检索结果-内蒙古大学图书馆

neural network language model Compression With Product Quantization and Soft Binarization

IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND language PROCESSING 2020年 28卷 2438-2449页

作者： Yu, Kai Ma, Rao Shi, Kaiyu Liu, Qi Shanghai Jiao Tong Univ AI Inst Dept Comp Sci & Engn SpeechLab Shanghai 200240 Peoples R China Shanghai Jiao Tong Univ AI Inst MoE Key Lab Artificial Intelligence Shanghai 200240 Peoples R China

Large memory consumption of the neural network language models (NN LMs) prohibits their use in many resource-constrained scenarios. Hence, effective NN LM compression approaches that are independent of NN structures are of great interest. However, previous approaches usually achieve a high compression ratio at the cost of obvious performance loss. In this paper, two recently proposed quantization approaches, product quantization (PQ) and soft binarization are effectively combined to address the issue. PQ decomposes word embedding matrices into a Cartesian product of low dimensional subspaces and quantizes each subspace separately. Soft binarization uses a small number of float scalars and the knowledge distillation technique to recover the performance loss during the binarization. Experiments show that the proposed approaches can achieve a high compression ratio, from 70 to over 100, while still maintaining comparable performance to the uncompressed NN LM on both PPL and word error rate criteria.

关键词： Artificial neural networks Quantization (signal) Matrix decomposition Vocabulary Memory management Logic gates neural network language model binarization product quantization

来源：评论

学校读者我要写书评

暂无评论

Class-Based neural network language model for Second-Pass Rescoring in ASR 22

Class-Based Neural Network Language Model for Second-Pass Re...

引用

Interspeech Conference

作者： Dai, Lingfeng Liu, Qi Yu, Kai Shanghai Jiao Tong Univ AI Inst Dept Comp Sci & Engn X LANCE Lab Shanghai Peoples R China

ISBN: (纸本)9781713836902

language model rescoring, especially neural network language model (NNLM) rescoring, is widely used to achieve improved performance in a second-pass automatic speech recognition (ASR) system. The rescoring NNLM is usually trained separately from the ASR system. Typically, the two's training corpora are different, leading to the vocabulary mismatch problem, consequently degrading ASR performance. Previous research focuses more on the language domain mismatch problem, while the vocabulary mismatch problem, which may also cause significant performance degradation, has not been well studied. This paper proposes a novel class-based NNLM framework to address the vocabulary mismatch problem for language model rescoring. Here, OOV words (unknown words to the rescoring NNLM are called OOV words for short) are assigned to well-trained classes of NNLM and inherit the class probability. Experiments show that class-based NNLM rescoring can significantly reduce performance degradation due to vocabulary mismatch.

关键词： neural network language model second-pass rescoring class-based softmax vocabulary mismatch parameter estimation

来源：评论

学校读者我要写书评

暂无评论

Multi-domain neural network language model

Multi-domain Neural Network Language Model

引用

14th Annual Conference of the International-Speech-Communication-Association (INTERSPEECH 2013)

作者： Alumae, Tanel Tallinn Univ Technol Inst Cybernet Tallinn Estonia

ISBN: (纸本)9781629934433

The paper describes a neural network language model that jointly models language in many related domains. In addition to the traditional layers of a neural network language model, the proposed model also trains a vector of factors for each domain in the training data that are used to modulate the connections from the projection layer to the hidden layer. The model is found to outperform simple neural network language models as well as domain-adapted maximum entropy language models in perplexity evaluation and speech recognition experiments.

关键词： neural network language model language model adaptation

来源：评论

学校读者我要写书评

暂无评论

Differentiable N-gram objective on abstractive summarization

引用

EXPERT SYSTEMS WITH APPLICATIONS 2023年 215卷

作者： Zhu, Yunqi Yang, Xuebing Wu, Yuanyuan Zhu, Mingjin Zhang, Wensheng Hainan Univ Sch Informat & Commun Engn Haikou Peoples R China Chinese Acad Sci Inst Automat Beijing Peoples R China South China Univ Technol Shien Ming Wu Sch Intelligent Engn Guangzhou Peoples R China

ROUGE is a standard automatic evaluation metric based on N-gram for sequence-to-sequence tasks like abstractive summarization, while cross-entropy loss is an essential objective that optimizes at unigram level for neural network language models. In this paper we present differentiable N-gram objectives, attempting to alleviate the discrepancy between training and evaluating criteria. The novelty of our work is the objective does not ceil the number of matched sub-sequences by the ground truth count of N-gram in reference sequence and weights the matched sub-sequences equally. Therefore, our proposed objective can maximize the probabilistic weight of matched sub-sequences. We jointly optimize cross-entropy loss and the objective, providing decent evaluation scores enhancement including ROUGE over abstractive summarization datasets CNN/DM and XSum, outperforming competitive N-gram objectives.

关键词： Abstractive summarization Differentiable N-gram objective neural network language model

来源：评论

学校读者我要写书评

暂无评论

Automatic Speech Recognition System with Output-Gate Projected Gated Recurrent Unit

引用

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS 2019年第2期E102D卷 355-363页

作者： Cheng, Gaofeng Zhang, Pengyuan Xu, Ji Univ Chinese Acad Sci Sch Elect Elect & Commun Engn Beijing Peoples R China Univ Chinese Acad Sci Beijing Peoples R China Chinese Acad Sci Inst Acoust Key Lab Speech Acoust & Content Understanding Beijing Peoples R China

The long short-term memory recurrent neural network (LSTM) has achieved tremendous success for automatic speech recognition (ASR). However, the complicated gating mechanism of LSTM introduces a massive computational cost and limits the application of LSTM in some scenarios. In this paper, we describe our work on accelerating the decoding speed and improving the decoding accuracy. First, we propose an architecture, which is called Projected Gated Recurrent Unit (PGRU), for ASR tasks, and show that the PGRU can consistently outperform the standard GRU. Second, to improve the PGRU generalization, particularly on large-scale ASR tasks, we propose the Output-gate PGRU (OPGRU). In addition, the time delay neural network (TDNN) and normalization methods are found beneficial for OPGRU. In this paper, we apply the OPGRU for both the acoustic model and recurrent neural network language model (RNN-LM). Finally, we evaluate the PGRU on the total Eval2000 / RT03 test sets, and the proposed OPGRU single ASR system achieves 0.9% / 0.9% absolute (8.2% / 8.6% relative) reduction in word error rate (WER) compared to our previous best LSTM single ASR system. Furthermore, the OPGRU ASR system achieves significant speed-up on both acoustic model and language model rescoring.

关键词： GRU LSTM neural network language model speech recognition

来源：评论

学校读者我要写书评

暂无评论

STATE-OF-THE-ART SPEECH RECOGNITION USING MULTI-STREAM SELF-ATTENTION WITH DILATED 1D CONVOLUTIONS

STATE-OF-THE-ART SPEECH RECOGNITION USING MULTI-STREAM SELF-...

引用

IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

作者： Han, Kyu J. Prieto, Ramon Ma, Tao ASAPP Inc Mountain View CA 94043 USA

ISBN: (纸本)9781728103068

Self-attention has been a huge success for many downstream tasks in NLP, which led to exploration of applying self-attention to speech problems as well. The efficacy of selfattention in speech applications, however, seems not fully blown yet since it is challenging to handle highly correlated speech frames in the context of self-attention. In this paper we propose a new neural network model architecture, namely multi-stream self-attention, to address the issue thus make the self-attention mechanism more effective for speech recognition. The proposed model architecture consists of parallel streams of self-attention encoders, and each stream has layers of 1D convolutions with dilated kernels whose dilation rates are unique given stream, followed by a self-attention layer. The self-attention mechanism in each stream pays attention to only one resolution of input speech frames and the attentive computation can be more efficient. In a later stage, outputs from all the streams are concatenated then linearly projected to the final embedding. By stacking the proposed multi-stream self-attention encoder blocks and rescoring the resultant lattices with neural network language models, we achieve the word error rate of 2.2% on the test-clean dataset of the LibriSpeech corpus, the best number reported thus far on the dataset.

关键词： Speech recognition multi-stream self-attention dilated 1D convolution neural network language model word error rate

来源：评论

学校读者我要写书评

暂无评论

Expanding Queries for Code Search Using Semantically Related API Class-names

引用

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 2018年第11期44卷 1070-1082页

作者： Zhang, Feng Niu, Haoran Keivanloo, Iman Zou, Ying Queens Univ Sch Comp Kingston ON K7L 3N6 Canada Queens Univ Dept Elect & Comp Engn Kingston ON K7L 3N6 Canada

When encountering unfamiliar programming tasks (e.g., connecting to a database), there is a need to seek potential working code examples. Instead of using code search engines, software developers usually post related programming questions on online Q&A forums (e.g., Stack Overflow). One possible reason is that existing code search engines would return effective code examples only if a query contains identifiers (e.g., class or method names). In other words, existing code search engines do not handle natural-language queries well (e.g., a description of a programming task). However, developers may not know the appropriate identifiers at the time of the search. As the demand of searching code examples is increasing, it is of significant interest to enhance code search engines. We conjecture that expanding natural-language queries with their semantically related identifiers has a great potential to enhance code search engines. In this paper, we propose an automated approach to find identifiers (in particular API class-names) that are semantically related to a given natural-language query. We evaluate the effectiveness of our approach using 74 queries on a corpus of 23,677,216 code snippets that are extracted from 24,666 open source Java projects. The results show that our approach can effectively recommend semantically related API class-names to expand the original natural-language queries. For instance, our approach successfully retrieves relevant code examples in the top 10 retrieved results for 76 percent of 74 queries, while it is 36 percent when using the original natural-language query;and the median rank of the first relevant code example is increased from 22 to 7.

关键词： Query expansion code search neural network language model API class-name

来源：评论

学校读者我要写书评

暂无评论

STRUCTURED OUTPUT LAYER neural network language model

STRUCTURED OUTPUT LAYER NEURAL NETWORK LANGUAGE MODEL

引用

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Le, Hai-Son Oparin, Ilya Allauzen, Alexandre Gauvain, Jean-Luc Yvon, Francois Univ Paris Sud BP 133 F-91403 Orsay France LIMIS CNRS F-91403 Orsay France

ISBN: (纸本)9781457705397

This paper introduces a new neural network language model (NNLM) based on word clustering to structure the output vocabulary: Structured Output Layer NNLM. This model is able to handle vocabularies of arbitrary size, hence dispensing with the design of short-lists that are commonly used in NNLMs. Several softmax layers replace the standard output layer in this model. The output structure depends on the word clustering which uses the continuous word representation induced by a NNLM. The GALE Mandarin data was used to carry out the speech-to-text experiments and evaluate the NNLMs. On this data the well tuned baseline system has a character error rate under 10%. Our model achieves consistent improvements over the combination of an n-gram model and classical short-list NNLMs both in terms of perplexity and recognition accuracy.

关键词： neural network language model Automatic Speech Recognition Speech-To-Text

来源：评论

学校读者我要写书评

暂无评论

Learning from past mistakes: improving automatic speech recognition output via noisy-clean phrase context modeling

引用

APSIPA TRANSACTIONS ON SIGNAL AND INFORMATION PROCESSING 2019年第1期8卷

作者： Shivakumar, Prashanth Gurunath Li, Haoqi Knight, Kevin Georgiou, Panayiotis Univ Southern Calif Signal Proc Commun Understanding & Behav Anal Lab Los Angeles CA 90007 USA Univ Southern Calif Informat Sci Inst Los Angeles CA USA

Automatic speech recognition (ASR) systems often make unrecoverable errors due to subsystem pruning (acoustic, language and pronunciation models);for example, pruning words due to acoustics using short-term context, prior to rescoring with long-term context based on linguistics. In this work, we model ASR as a phrase-based noisy transformation channel and propose an error correction system that can learn from the aggregate errors of all the independent modules constituting the ASR and attempt to invert those. The proposed system can exploit long-term context using a neural network language model and can better choose between existing ASR output possibilities as well as re-introduce previously pruned or unseen (Out-Of-Vocabulary) phrases. It provides corrections under poorly performing ASR conditions without degrading any accurate transcriptions;such corrections are greater on top of out-of-domain andmismatched data ASR. Our systemconsistently provides improvements over the baseline ASR, even when baseline is further optimized through Recurrent neural network (RNN) languagemodel rescoring. This demonstrates that any ASR improvements can be exploited independently and that our proposed system can potentially still provide benefits on highly optimized ASR. Finally, we present an extensive analysis of the type of errors corrected by our system.

关键词： Error correction Speech recognition Phrase-based context modeling Noise channel estimation neural network language model

来源：评论

学校读者我要写书评

暂无评论

Comparison of Various neural network language models in Speech Recognition 3

Comparison of Various Neural Network Language Models in Spee...

引用

3rd International Conference on Information Science and Control Engineering (ICISCE)

作者： Zuo, Lingyun Liu, Jian Wan, Xin IACAS Key Lab Speech Acoust & Content Beijing Peoples R China Chinese Acad Sci XTIPC Xinjiang Lab Minor Speech & Language Informat Pro Beijing Peoples R China Natl Comp Network Emergency Response Tech Team Coordinat Ctr Beijing Peoples R China

ISBN: (纸本)9781509025350

In recent years, research on language modeling for speech recognition has increasingly focused on the application of neural networks. However, the performance of neural network language models strongly depends on their architectural structure. Three competing concepts have been developed: Firstly, feed forward neural networks representing an n-gram approach;Secondly, recurrent neural networks that may learn context dependencies spanning more than a fixed number of predecessor words;Thirdly, the long short-term memory (LSTM) neural networks can fully exploits the correlation on a telephone conversation corpus. In this paper, we compare count models to feed forward, recurrent, and LSTM neural network in conversational telephone speech recognition tasks. Furthermore, we put forward a language model estimation method introduced the information of history sentences. We evaluate the models in terms of perplexity and word error rate, experimentally validating the strong correlation of the two quantities, which we find to hold regardless of the underlying type of the language model. The experimental results show that the performance of LSTM neural network language model is optimal in n-best lists rescore. Compared to the first pass decoding, the relative decline in average word error rate is 4.3% when using ten candidate results to re-score in conversational telephone speech recognition tasks.

关键词： neural network language model LSTM speech recognition n-best lists re-score

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：