检索结果-内蒙古大学图书馆

Frustratingly easy noise-aware training of acoustic models

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Raj, Desh Villalba, Jesús Povey, Daniel Khudanpur, Sanjeev Center for Language and Speech Processing & Human Language Technology Center of Excellence The Johns Hopkins University BaltimoreMD21218 United States Xiaomi Corp. Beijing China

Environmental noises and reverberation have a detrimental effect on the performance of automatic speech recognition (ASR) systems. Multi-condition training of neural network-based acoustic models is used to deal with this problem, but it requires many-folds data augmentation, resulting in increased training time. In this paper, we propose utterance-level noise vectors for noise-aware training of acoustic models in hybrid ASR. Our noise vectors are obtained by combining the means of speech frames and silence frames in the utterance, where the speech/silence labels may be obtained from a GMM-HMM model trained for ASR alignments, such that no extra computation is required beyond averaging of feature vectors. We show through experiments on AMI and Aurora-4 that this simple adaptation technique can result in 6-7% relative WER improvement. We implement several embedding-based adaptation baselines proposed in literature, and show that our method outperforms them on both the datasets. Finally, we extend our method to the online ASR setting by using frame-level maximum likelihood for the mean estimation. © 2020, CC BY.

关键词： speech recognition

Mixture of speaker-type PLDAs for children's speech diarization

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Xie, Jiamin Sia, Suzanna García, Paola Povey, Daniel Khudanpur, Sanjeev Center for Language and Speech Processing Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD21218 United States Xiaomi Corp. Beijing China

In diarization, the PLDA is typically used to model an inference structure which assumes the variation in speech segments be induced by various speakers. The speaker variation is then learned from the training data. However, human perception can differentiate speakers by age, gender, among other characteristics. In this paper, we investigate a speaker-type informed model that explicitly captures the known variation of speakers. We explore a mixture of three PLDA models, where each model represents an adult female, male, or child category. The weighting of each model is decided by the prior probability of its respective class, which we study. The evaluation is performed on a subset of the BabyTrain corpus. We examine the expected performance gain using the oracle speaker type labels, which yields an 11.7% DER reduction. We introduce a novel baby vocalization augmentation technique and then compare the mixture model to the single model. Our experimental result shows an effective 0.9% DER reduction obtained by adding vocalizations. We discover empirically that a balanced dataset is important to train the mixture PLDA model, which outperforms the single PLDA by 1.3% using the same training data and achieving a 35.8% DER. The same setup improves over a standard baseline by 2.8% DER. Index Terms: speaker diarization, children’s speech, transformer encoder, mixture of PLDAs Copyright © 2020, The Authors. All rights reserved.

关键词： Mixtures

Neural language modeling with implicit cache pointers

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Li, Ke Povey, Daniel Khudanpur, Sanjeev Center for Language and Speech Processing Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD21218 United States Xiaomi Corp. Beijing China

A cache-inspired approach is proposed for neural language models (LMs) to improve long-range dependency and better predict rare words from long contexts. This approach is a simpler alternative to attention-based pointer mechanism that enables neural LMs to reproduce words from recent history. Without using attention and mixture structure, the method only involves appending extra tokens that represent words in history to the output layer of a neural LM and modifying training supervisions accordingly. A memory-augmentation unit is introduced to learn words that are particularly likely to repeat. We experiment with both recurrent neural network- and Transformer-based LMs. Perplexity evaluation on Penn Treebank and WikiText-2 shows the proposed model outperforms both LSTM and LSTM with attention-based pointer mechanism and is more effective on rare words. N-best rescoring experiments on Switchboard indicate that it benefits both very rare and frequent words. However, it is challenging for the proposed model as well as two other models with attention-based pointer mechanism to obtain good overall WER reductions. Copyright © 2020, The Authors. All rights reserved.

关键词： speech recognition

SPECTRE: Visual speech-Informed Perceptual 3D Facial Expression Reconstruction from Videos

学校读者我要写书评

暂无评论

SPECTRE: Visual Speech-Informed Perceptual 3D Facial Express...

IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

作者： Panagiotis P. Filntisis George Retsinas Foivos Paraperas-Papantoniou Athanasios Katsamanis Anastasios Roussos Petros Maragos Institute of Robotics Athena Research Center Maroussi Greece School of Electrical & Computer Engineering National Technical University of Athens Greece Imperial College London UK Institute for Language and Speech Processing Athena R.C. Greece Institute of Computer Science (ICS) Foundation for Research & Technology - Hellas (FORTH) Greece College of Engineering Mathematics and Physical Sciences University of Exeter UK

The recent state of the art on monocular 3D face reconstruction from image data has made some impressive advancements, thanks to the advent of Deep Learning. However, it has mostly focused on input coming from a single RGB image, overlooking the following important factors: a) Nowadays, the vast majority of facial image data of interest do not originate from single images but rather from videos, which contain rich dynamic information. b) Furthermore, these videos typically capture individuals in some form of verbal communication (public talks, teleconferences, audiovisual human-computer interactions, interviews, monologues/dialogues in movies, etc). When existing 3D face reconstruction methods are applied in such videos, the artifacts in the reconstruction of the shape and motion of the mouth area are often severe, since they do not match well with the speech *** overcome the aforementioned limitations, we present the first method for visual speech-informed perceptual reconstruction of 3D mouth expressions. We do this by proposing a "lipreading" loss, which guides the fitting process so that the elicited perception from the 3D reconstructed talking head resembles that of the original video footage. We demonstrate that, interestingly, the lipreading loss is better suited for 3D reconstruction of mouth movements compared to traditional landmark losses, and even direct 3D supervision. Furthermore, the devised method does not rely on any text transcriptions or corresponding audio, rendering it ideal for training in unlabeled datasets. We verify the efficiency of our method through objective evaluations on three large-scale datasets, as well as subjective evaluation with two web-based user studies. Project webpage: https://***/spectre/

关键词：

Target-speaker voice activity detection with improved i-vector estimation for unknown number of speaker

学校读者我要写书评

暂无评论

arXiv 2021年

作者： He, Maokui Raj, Desh Huang, Zili Du, Jun Chen, Zhuo Watanabe, Shinji University of Science and Technology of China HeFei China Center for Language and Speech Processing The Johns Hopkins University BaltimoreMD United States Microsoft Corp RedmondWA United States

Target-speaker voice activity detection (TS-VAD) has recently shown promising results for speaker diarization on highly overlapped speech. However, the original model requires a fixed (and known) number of speakers, which limits its application to real conversations. In this paper, we extend TS-VAD to speaker diarization with unknown numbers of speakers. This is achieved by two steps: first, an initial diarization system is applied for speaker number estimation, followed by TS-VAD network output masking according to this estimate. We further investigate different diarization methods, including clustering-based and region proposal networks, for estimating the initial i-vectors. Since these systems have complementary strengths, we propose a fusion-based method to combine frame-level decisions from the systems for an improved initialization. We demonstrate through experiments on variants of the LibriCSS meeting corpus that our proposed approach can improve the DER by up to 50% relative across varying numbers of speakers. This improvement also results in better downstream ASR performance approaching that using oracle segments. © 2021, CC0.

关键词： speech recognition

Align or attend? Toward More Efficient and Accurate Spoken Word Discovery Using speech-to-Image Retrieval

学校读者我要写书评

暂无评论

Align or attend? Toward More Efficient and Accurate Spoken W...

IEEE International Conference on Acoustics, speech and Signal processing

作者： Liming Wang Xinsheng Wang Mark Hasegawa-Johnson Odette Scharenborg Najim Dehak University of Illinois at Urbana-Champaign School of Software Engineering Xi’an Jiaotong University Multimedia Computing Group Delft University of Technology Center for Language and Speech Processing Johns Hopkins University

ISBN: (纸本)9781728176055;9781728176062

Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.

关键词： Visualization Conferences Signal processing Acoustics speech processing

Training noisy single-channel speech separation with noisy oracle sources: A large gap and a small step

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Maciejewski, Matthew Shi, Jing Watanabe, Shinji Khudanpur, Sanjeev Center for Language and Speech Processing The Johns Hopkins University United States Human Language Technology Center of Excellence The Johns Hopkins University United States Institute of Automation Chinese Academy of Sciences China

As the performance of single-channel speech separation systems has improved, there has been a desire to move to more challenging conditions than the clean, near-field speech that initial systems were developed on. When training deep learning separation models, a need for ground truth leads to training on synthetic mixtures. As such, training in noisy conditions requires either using noise synthetically added to clean speech, preventing the use of in-domain data for a noisy-condition task, or training using mixtures of noisy speech, requiring the network to additionally separate the noise. We demonstrate the relative inseparability of noise and that this noisy speech paradigm leads to significant degradation of system performance. We also propose an SI-SDR–inspired training objective that tries to exploit the inseparability of noise to implicitly partition the signal and discount noise separation errors, enabling the training of better separation systems with noisy oracle sources. Copyright © 2020, The Authors. All rights reserved.

关键词： speech analysis

Dover-lap: A method for combining overlap-aware diarization outputs

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Raj, Desh Garcia-Perera, Leibny Paola Huang, Zili Watanabe, Shinji Povey, Daniel Stolcke, Andreas Khudanpur, Sanjeev Center for Language and Speech Processing & Human Language Technlogy Center of Excellence The Johms Hopkins University BaltimoreMD21218 United States Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD21218 United States Xiaomi Corp. Beijing China Amazon Alexa Speech SunnyvaleCA United States

Several advances have been made recently towards handling overlapping speech for speaker diarization. Since speech and natural language tasks often benefit from ensemble techniques, we propose an algorithm for combining outputs from such diarization systems through majority voting. Our method, DOVER-Lap, is inspired from the recently proposed DOVER algorithm, but is designed to handle overlapping segments in diarization outputs. We also modify the pair-wise incremental label mapping strategy used in DOVER, and propose an approximation algorithm based on weighted k-partite graph matching, which performs this mapping using a global cost tensor. We demonstrate the strength of our method by combining outputs from diverse systems — clustering-based, region proposal networks, and target-speaker voice activity detection — on AMI and LibriCSS datasets, where it consistently outperforms the single best system. Additionally, we show that DOVER-Lap can be used for late fusion in multichannel diarization, and compares favorably with early fusion methods like beamforming. © 2020, CC-BY.

关键词： Approximation algorithms

SAV-SE: Scene-aware Audio-Visual speech Enhancement with Selective State Space Model

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Qian, Xinyuan Gao, Jiaran Zhang, Yaodan Zhang, Qiquan Liu, Hexin Garcia, Leibny Paola Li, Haizhou School of Computer and Communication Engineering University of Science and Technology Beijing Beijing100083 China School of Electrical Engineering and Telecommunications The University of New South Wales Sydney2052 Australia College of Computing and Data Science Nanyang Technological University Singapore Center for Language and Speech Processing Johns Hopkins University United States Guangdong Provincial Key Laboratory of Big Data Computing The Chinese University of Hong Kong Shenzhen518172 China Shenzhen Research Institute of Big data Shenzhen51872 China

speech enhancement plays an essential role in various applications, and the integration of visual information has been demonstrated to bring substantial advantages. However, the majority of current research concentrates on the examination of facial and lip movements, which can be compromised or entirely inaccessible in scenarios where occlusions occur or when the camera view is distant. Whereas contextual visual cues from the surrounding environment have been overlooked: for example, when we see a dog bark, our brain has the innate ability to discern and filter out the barking noise. To this end, in this paper, we introduce a novel task, i.e. Scene-aware AudioVisual speech Enhancement (SAV-SE). To our best knowledge, this is the first proposal to use rich contextual information from synchronized video as auxiliary cues to indicate the type of noise, which eventually improves the speech enhancement performance. Specifically, we propose the VC-S2E method, which incorporates the Conformer and Mamba modules for their complementary strengths. Extensive experiments are conducted on public MUSIC, AVspeech and AudioSet datasets, where the results demonstrate the superiority of VC-S2E over other competitive methods. We will make the source code publicly available. Project demo page: https://***/ © 2024, CC BY.

关键词： speech enhancement