检索结果-内蒙古大学图书馆

Quality-Aware End-to-End Audio-Visual Neural Speaker Diarization

学校读者我要写书评

暂无评论

arXiv 2024年

作者： He, Mao-Kui Du, Jun Niu, Shu-Tong Liu, Qing-Feng Lee, Chin-Hui National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Anhui Hefei China IFlytek Hefei Anhui China School of Electrical and Computer Engineering Georgia Institute of Technology AtlantaGA United States

In this paper, we propose a quality-aware end-to-end audio-visual neural speaker diarization framework, which comprises three key techniques. First, our audio-visual model takes both audio and visual features as inputs, utilizing a series of binary classification output layers to simultaneously identify the activities of all speakers. This end-to-end framework is meticulously designed to effectively handle situations of overlapping speech, providing accurate discrimination between speech and non-speech segments through the utilization of multimodal information. Next, we employ a quality-aware audio-visual fusion structure to address signal quality issues for both audio degradations, such as noise, reverberation and other distortions, and video degradations, such as occlusions, off-screen speakers, or unreliable detection. Finally, a cross attention mechanism applied to multi-speaker embedding empowers the network to handle scenarios with varying numbers of speakers. Our experimental results, obtained from various data sets, demonstrate the robustness of our proposed techniques in diverse acoustic environments. Even in scenarios with severely degraded video quality, our system attains performance levels comparable to the best available audio-visual systems. Copyright © 2024, The Authors. All rights reserved.

关键词： Reverberation

Wav2f0: Exploring the Potential of Wav2vec 2.0 for speech Fundamental Frequency Extraction

学校读者我要写书评

暂无评论

Wav2f0: Exploring the Potential of Wav2vec 2.0 for Speech Fu...

International Symposium on Chinese Spoken language processing

作者： Rui Feng Yin-Long Liu Zhen-Hua Ling Jia-Hong Yuan National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei P. R. China Interdisciplinary Research Center for Linguistic Sciences University of Science and Technology of China Hefei P. R. China

ISBN: (数字)9798331516826

ISBN: (纸本)9798331516833

speech fundamental frequency (F0) extraction is one of the most important tasks in speech signal processing. This paper aims to explore the feasibility of using deep learning for speech fundamental frequency extraction. Our approach, Wav2f0, combines the Wav2vec 2.0 model with fully connected and LSTM layers, leveraging the pretrained representations learned by Wav2vec 2.0. We conduct training and evaluation on a Vietnamese tone production corpus, which contains parallel recordings of Electroglottograph (EGG) and microphone signals. Wav2f0 outperforms Praat in pitch extraction accuracy on the corpus, especially in scenarios where Praat fails to estimate pitch. The Gross Pitch Error (GPE) of Wav2f0 is 7.1%, representing a more than 50% error reduction compared to Praat's 15.5%.

关键词： Deep learning Training Measurement Production Feature extraction Recording Multilingual speech processing Long short term memory Microphones

SHINE: Syntax-augmented Hierarchical Interactive Encoder for Zero-shot Cross-lingual Information Extraction

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ma, Jun-Yu Gu, Jia-Chen Ling, Zhen-Hua Liu, Quan Liu, Cong Hu, Guoping National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China State Key Laboratory of Cognitive Intelligence China iFLYTEK Research Hefei China

Zero-shot cross-lingual information extraction (IE) aims at constructing an IE model for some low-resource target languages, given annotations exclusively in some rich-resource languages. Recent studies based on language-universal features have shown their effectiveness and are attracting increasing attention. However, prior work has neither explored the potential of establishing interactions between language-universal features and contextual representations nor incorporated features that can effectively model constituent span attributes and relationships between multiple spans. In this study, a syntax-augmented hierarchical interactive encoder (SHINE) is proposed to transfer cross-lingual IE knowledge. The proposed encoder is capable of interactively capturing complementary information between features and contextual information, to derive language-agnostic representations for various IE tasks. Concretely, a multilevel interaction network is designed to hierarchically interact the complementary information to strengthen domain adaptability. Besides, in addition to the well-studied syntax features of part-of-speech and dependency relation, a new syntax feature of constituency structure is introduced to model the constituent span information which is crucial for IE. Experiments across seven languages on three IE tasks and four benchmarks verify the effectiveness and generalization ability of the proposed method. Copyright © 2023, The Authors. All rights reserved.

关键词： Information retrieval

GIFT: Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Gu, Jia-Chen Ling, Zhen-Hua Liu, Quan Liu, Cong Hu, Guoping National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China State Key Laboratory of Cognitive Intelligence China iFLYTEK Research Hefei China

Addressing the issues of who saying what to whom in multi-party conversations (MPCs) has recently attracted a lot of research attention. However, existing methods on MPC understanding typically embed interlocutors and utterances into sequential information flows, or utilize only the superficial of inherent graph structures in MPCs. To this end, we present a plug-and-play and lightweight method named graph-induced fine-tuning (GIFT) which can adapt various Transformer-based pre-trained language models (PLMs) for universal MPC understanding. In detail, the full and equivalent connections among utterances in regular Transformer ignore the sparse but distinctive dependency of an utterance on another in MPCs. To distinguish different relationships between utterances, four types of edges are designed to integrate graph-induced signals into attention mechanisms to refine PLMs originally designed for processing sequential texts. We evaluate GIFT by implementing it into three PLMs, and test the performance on three downstream tasks including addressee recognition, speaker identification and response selection. Experimental results show that GIFT can significantly improve the performance of three PLMs on three downstream tasks and two benchmarks with only 4 additional parameters per encoding layer, achieving new state-of-the-art performance on MPC understanding. Copyright © 2023, The Authors. All rights reserved.

关键词： Benchmarking

TEGTOK: Augmenting Text Generation via Task-specific and Open-world Knowledge

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Tan, Chao-Hong Gu, Jia-Chen Tao, Chongyang Ling, Zhen-Hua Xu, Can Hu, Huang Geng, Xiubo Jiang, Daxin National Engineering Research Center for Speech and Language Information Processing University of Science and Technology of China Hefei China Microsoft Beijing China

Generating natural and informative texts has been a long-standing problem in NLP. Much effort has been dedicated into incorporating pre-trained language models (PLMs) with various open-world knowledge, such as knowledge graphs or wiki pages. However, their ability to access and manipulate the task-specific knowledge is still limited on downstream tasks, as this type of knowledge is usually not well covered in PLMs and is hard to acquire. To address the problem, we propose augmenting TExt Generation via Task-specific and Open-world Knowledge (TEGTOK) in a unified framework. Our model selects knowledge entries from two types of knowledge sources through dense retrieval and then injects them into the input encoding and output decoding stages respectively on the basis of PLMs. With the help of these two types of knowledge, our model can learn what and how to generate. Experiments on two text generation tasks of dialogue generation and question generation, and on two datasets show that our method achieves better performance than various baseline models. Copyright © 2022, The Authors. All rights reserved.

关键词： Knowledge graph

Focus on the Present: A Regularization Method for the ASR Source-Target Attention Layer

学校读者我要写书评

暂无评论

Focus on the Present: A Regularization Method for the ASR So...

International Conference on Acoustics, speech, and Signal processing (ICASSP)

作者： Nanxin Chen Piotr Żelasko Jesús Villalba Najim Dehak Center for Language and Speech Processing Johns Hopkins University Baltimore MD Human Language Technology Center of Excellence Johns Hopkins University Baltimore MD

This paper introduces a novel method to diagnose the source-target attention in state-of-the-art end-to-end speech recognition models with joint connectionist temporal classification (CTC) and attention training. Our method is based on the fact that both, CTC and source-target attention, are acting on the same encoder representations. To understand the functionality of the attention, CTC is applied to compute the token posteriors given the attention outputs. We found that the source-target attention heads are able to predict several tokens ahead of the current one. Inspired by the observation, a new regularization method is proposed which leverages CTC to make source-target attention more focused on the frames corresponding to the output token being predicted by the decoder. Experiments reveal stable improvements up to 7% and 13% relatively with the proposed regularization on TED-LIUM 2 and Librispeech.

关键词： Training Adaptation models Conferences Computational modeling speech recognition Signal processing Acoustics

Injecting Text and Cross-lingual Supervision in Few-shot Learning from Self-Supervised Models

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Wiesner, Matthew Raj, Desh Khudanpur, Sanjeev Human Language Technology Center of Excellence Johns Hopkins University United States Center for Language and Speech Processing Johns Hopkins University United States

Self-supervised model pre-training has recently garnered significant interest, but relatively few efforts have explored using additional resources in fine-tuning these models. We demonstrate how universal phoneset acoustic models can leverage cross-lingual supervision to improve transfer of pretrained self-supervised representations to new languages. We also show how target-language text can be used to enable and improve fine-tuning with the lattice-free maximum mutual information (LF-MMI) objective. In three low-resource languages these techniques greatly improved few-shot learning performance. © 2021, CC BY.

关键词： Learning systems

Two-stage augmentation and adaptive CTC fusion for improved robustness of multi-stream end-to-end ASR

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Li, Ruizhi Sell, Gregory Hermansky, Hynek Center for Language and Speech Processing Johns Hopkins University United States Human Language Technology Center of Excellence Johns Hopkins University United States

Performance degradation of an Automatic speech Recognition (ASR) system is commonly observed when the test acoustic condition is different from training. Hence, it is essential to make ASR systems robust against various environmental distortions, such as background noises and reverberations. In a multi-stream paradigm, improving robustness takes account of handling a variety of unseen single-stream conditions and inter-stream dynamics. Previously, a practical two-stage training strategy was proposed within multi-stream end-to-end ASR, where Stage-2 formulates the multi-stream model with features from Stage-1 Universal Feature Extractor (UFE). In this paper, as an extension, we introduce a two-stage augmentation scheme focusing on mismatch scenarios: Stage-1 Augmentation aims to address single-stream input varieties with data augmentation techniques;Stage-2 Time Masking applies temporal masks on UFE features of randomly selected streams to simulate diverse stream combinations. During inference, we also present adaptive Connectionist Temporal Classification (CTC) fusion with the help of hierarchical attention mechanisms. Experiments have been conducted on two datasets, DIRHA and AMI, as a multi-stream scenario. Compared with the previous training strategy, substantial improvements are reported with relative word error rate reductions of 29.7 − 59.3% across several unseen stream combinations. © 2021, CC BY.

关键词： speech recognition

Radically old way of computing spectra: Applications in end-to-end ASR

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Sadhu, Samik Hermansky, Hynek Center for Language and Speech Processing Johns Hopkins University United States Human Language Technology Center of Excellence Johns Hopkins University United States

We propose a technique to compute spectrograms using Frequency Domain Linear Prediction (FDLP) that uses all-pole models to fit the squared Hilbert envelope of speech in different frequency sub-bands. The spectrogram of a complete speech utterance is computed by overlap-add of contiguous all-pole model responses. A long context window of 1.5 seconds allows us to capture the low frequency temporal modulations of speech in the spectrogram. For an end-to-end automatic speech recognition task, the FDLP spectrogram performs on par with the standard mel spectrogram features for clean read speech training and test data. For more realistic speech data with train-test domain mismatches or reverberations, FDLP spectrogram shows up to 25% and 22% relative WER improvements over mel spectrogram respectively. Copyright © 2021, The Authors. All rights reserved.

关键词： speech recognition