检索结果-内蒙古大学图书馆

作者： Ai, Yang Ling, Zhen-Hua Wu, Wei-Lu Li, Ang University of Science and Technology of China National Engineering Research Center of Speech and Language Information Processing Hefei230027 China National University of Defense Technology Hefei230037 China

This paper presents a denoising and dereverberation hierarchical neural vocoder (DNR-HiNet) to convert noisy and reverberant acoustic features into clean speech waveforms. The DNR-HiNet vocoder is built by modifying the amplitude spectrum predictor (ASP) in the original HiNet vocoder. This modified denoising and dereverberation ASP (DNR-ASP) can predict clean log amplitude spectra from input degraded acoustic features. To achieve this, the DNR-ASP first predicts the log amplitude spectra of noisy and reverberant speech, the log amplitude spectra of additive noise and the room impulse response (RIR) and then performs initial denoising and dereverberation by signal processing algorithms. The initially processed log amplitude spectra are then enhanced by another neural network to obtain the final clean log amplitude spectra. We also introduce a bandwidth extension model and a frequency resolution extension model into the DNR-ASP to further improve its performance. Finally, a statistical parametric speech synthesis (SPSS) method with DNR-HiNet is proposed to deal with the situation that the quality of target speaker's recordings is degraded by noise and reverberation. Experimental results indicate that the DNR-HiNet vocoder was able to generate denoised and dereverberated waveforms given noisy and reverberant acoustic features and outperformed the original HiNet vocoder and a few other neural vocoders. On speech enhancement tasks, its performance was competitive with several advanced speech enhancement methods. Furthermore, the SPSS method with DNR-HiNet achieved better quality of synthetic speech than the conventional approach which directly applied speech enhancement to the degraded adaptation data. © 2014 IEEE.

关键词： Hidden Markov models

来源：评论

学校读者我要写书评

暂无评论

Lightweight Audio-Visual Wake Word Spotting with Diverse Acoustic Knowledge Distillation

引用

IEEE Transactions on Circuits and Systems for Video Technology 2025年

作者： Li, Ke-Wei Chen, Hang Du, Jun Zhou, Heng-Shun Siniscalchi, Sabato Marco Niu, Shu-Tong Xiong, Shi-Fu University of Science and Technology of China National Engineering Research Center of Speech and Language Information Processing Anhui Hefei China IFlytek Research Anhui Hefei China University of Palermo Italy

Audio-Visual Wake Word Spotting (AVWWS) aims to accurately detect user-defined keywords by leveraging the complementary nature of different modalities in challenging acoustic environments. However, two primary challenges hinder the application of AVWWS models in real-world scenarios: increased model parameters involving the video modality and the scarcity of paired audio-visual data. To address these issues, we propose a novel diverse acoustic knowledge distillation (DAKD) framework, which utilizes easily accessible single-modality audio data to train two teacher models and employs cross-modal knowledge distillation to transfer the generalization and denoising capabilities of the teachers to the audio-visual student model. This approach mitigates the overfitting risk associated with large parameter counts and limited data. The DAKD framework consists of an audio-visual student model based on the lightweight multi-scale temporal-spatial attention (LMTSA) architecture, a multi-conditional teacher (MCT) model, and a de-noising teacher (DNT) model. The LMTSA model integrates compact 3D and 2D blocks based on the ResNet architecture through a simple attention module and accepts multi-scale supervision from word-level and phone-level labels, achieving joint temporal-spatial modeling with minimal parameter usage. The MCT and DNT models were trained using extensive real or simulated far-field speech and paired near-field and far-field speech, respectively, to generalize unseen acoustic environments and de-noising capabilities to the audio-visual student model. The effectiveness of our proposed DAKD framework is validated through comprehensive experiments on the MISP2021 and the updated MISP2021 Eval Hard datasets, establishing new benchmarks with fewer parameters. Our code will be available at https://***/wikkk-tp/AVWWS_DAKD. © 1991-2012 IEEE.

关键词： Students

来源：评论

学校读者我要写书评

暂无评论

MLCA-AVSR: Multi-Layer Cross Attention Fusion Based Audio-Visual speech Recognition

MLCA-AVSR: Multi-Layer Cross Attention Fusion Based Audio-Vi...

引用

International Conference on Acoustics, speech, and Signal processing (ICASSP)

作者： He Wang Pengcheng Guo Pan Zhou Lei Xie Audio Speech and Language Processing Group (ASLP@NPU) School of Computer Science Northwestern Polytechnical University Xian China Space AI Li Auto

While automatic speech recognition (ASR) systems degrade significantly in noisy environments, audio-visual speech recognition (AVSR) systems aim to complement the audio stream with noise-invariant visual cues and improve the system’s robustness. However, current studies mainly focus on fusing the well-learned modality features, like the output of modality-specific encoders, without considering the contextual relationship during the modality feature learning. In this study, we propose a multi-layer cross-attention fusion based AVSR (MLCA-AVSR) approach that promotes representation learning of each modality by fusing them at different levels of audio/visual encoders. Experimental results on the MISP2022-AVSR Challenge dataset show the efficacy of our proposed system, achieving a concatenated minimum permutation character error rate (cpCER) of 30.57% on the Eval set and yielding up to 3.17% relative improvement compared with our previous system which ranked the second place in the challenge. Following the fusion of multiple systems, our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Voice Attribute Editing with Text Prompt

arXiv

引用

arXiv 2024年

作者： Sheng, Zhengyan Ai, Yang Liu, Li-Juan Pan, Jia Ling, Zhen-Hua National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China iFLYTEK Research Hefei China

Despite recent advancements in speech generation with text prompt providing control over speech style, voice attributes in synthesized speech remain elusive and challenging to control. This paper introduces a novel task: voice attribute editing with text prompt, with the goal of making relative modifications to voice attributes according to the actions described in the text prompt. To solve this task, VoxEditor, an end-to-end generative model, is proposed. In VoxEditor, addressing the insufficiency of text prompt, a Residual Memory (ResMem) block is designed, that efficiently maps voice attributes and these descriptors into the shared feature space. Additionally, the ResMem block is enhanced with a voice attribute degree prediction (VADP) block to align voice attributes with corresponding descriptors, addressing the imprecision of text prompt caused by non-quantitative descriptions of voice attributes. We also establish the open-source VCTK-RVA dataset, which leads the way in manual annotations detailing voice characteristic differences among different speakers. Extensive experiments demonstrate the effectiveness and generalizability of our proposed method in terms of both objective and subjective metrics. The dataset and audio samples are available on the website 1 © 2024, CC BY.

关键词： Open systems

来源：评论

学校读者我要写书评

暂无评论

APNet2: High-quality and High-efficiency Neural Vocoder with Direct Prediction of Amplitude and Phase Spectra

arXiv

引用

arXiv 2023年

作者： Du, Hui-Peng Lu, Ye-Xin Ai, Yang Ling, Zhen-Hua National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China

In our previous work, we proposed a neural vocoder called APNet, which directly predicts speech amplitude and phase spectra with a 5 ms frame shift in parallel from the input acoustic features, and then reconstructs the 16 kHz speech waveform using inverse short-time Fourier transform (ISTFT). APNet demonstrates the capability to generate synthesized speech of comparable quality to the HiFi-GAN vocoder but with a considerably improved inference speed. However, the performance of the APNet vocoder is constrained by the waveform sampling rate and spectral frame shift, limiting its practicality for high-quality speech synthesis. Therefore, this paper proposes an improved iteration of APNet, named APNet2. The proposed APNet2 vocoder adopts ConvNeXt v2 as the backbone network for amplitude and phase predictions, expecting to enhance the modeling capability. Additionally, we introduce a multi-resolution discriminator (MRD) into the GAN-based losses and optimize the form of certain losses. At a common configuration with a waveform sampling rate of 22.05 kHz and spectral frame shift of 256 points (i.e., approximately 11.6ms), our proposed APNet2 vocoder outperformed the original APNet and Vocos vocoders in terms of synthesized speech quality. The synthesized speech quality of APNet2 is also comparable to that of HiFi-GAN and iSTFTNet, while offering a significantly faster inference speed. Copyright © 2023, The Authors. All rights reserved.

关键词： Discriminators

来源：评论

学校读者我要写书评

暂无评论

Optimizing Dysarthria Wake-Up Word Spotting: an End-to-End Approach For SLT 2024 LRDWWS Challenge

Optimizing Dysarthria Wake-Up Word Spotting: an End-to-End A...

引用

IEEE Spoken language Technology Workshop

作者： Shuiyun Liu Yuxiang Kong Pengcheng Guo Weiji Zhuang Peng Gao Yujun Wang Lei Xie Audio Speech and Language Processing Group (ASLP@NPU) School of Computer Science Northwestern Polytechnical University Xi’an China Xiaomi Inc. China

ISBN: (数字)9798350392258

ISBN: (纸本)9798350392265

speech has emerged as a widely embraced user interface across diverse applications. However, for individuals with dysarthria, the inherent variability in their speech poses significant challenges. This paper presents an end-to-end Pretrain-based Dual-filter Dysarthria Wake-up word Spotting (PD-DWS) system for the SLT 2024 Low-Resource Dysarthria Wake-Up Word Spotting Challenge. Specifically, our system improves performance from two key perspectives: audio modeling and dual-filter strategy. For audio modeling, we propose an innovative 2 branch- d 2 v 2 model based on the pre-trained data2vec $2(\mathrm{~d} 2 \mathrm{v} 2)$, which can simultaneously model automatic speech recognition (ASR) and wake-up word spotting (WWS) tasks through a unified multi-task finetuning paradigm. Additionally, a dual-filter strategy is introduced to reduce the false accept rate (FAR) while maintaining the same false reject rate (FRR). Experimental results demonstrate that our PD-DWS system achieves an FAR of 0.00321 and an FRR of 0.005, with a total score of 0.00821 on the test-B eval set, securing first place in the challenge.

关键词： Conferences User interfaces Multitasking Data augmentation Automatic speech recognition

来源：评论

学校读者我要写书评

暂无评论

Stage-Wise and Prior-Aware Neural speech Phase Prediction

Stage-Wise and Prior-Aware Neural Speech Phase Prediction

引用

IEEE Spoken language Technology Workshop

作者： Fei Liu Yang Ai Hui-Peng Du Ye-Xin Lu Rui-Chen Zheng Zhen-Hua Ling National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei P. R. China

ISBN: (数字)9798350392258

ISBN: (纸本)9798350392265

This paper proposes a novel Stage-wise and Prior-aware Neural speech Phase Prediction (SP-NSPP) model, which predicts the phase spectrum from input amplitude spectrum by two-stage neural networks. In the initial prior-construction stage, we preliminarily predict a rough prior phase spectrum from the amplitude spectrum. The subsequent refinement stage transforms the amplitude spectrum into a refined high-quality phase spectrum conditioned on the prior phase. Networks in both stages use ConvNeXt v2 blocks as the backbone and adopt adversarial training by innovatively introducing a phase spectrum discriminator (PSD). To further improve the continuity of the refined phase, we also incorporate a time-frequency integrated difference (TFID) loss in the refinement stage. Experimental results confirm that, compared to neural network-based no-prior phase prediction methods, the proposed SP-NSPP achieves higher phase prediction accuracy, thanks to introducing the coarse phase priors and diverse training criteria. Compared to iterative phase estimation algorithms, our proposed SP-NSPP does not require multiple rounds of staged iterations, resulting in higher generation efficiency.

关键词： Training Time-frequency analysis Accuracy Phase estimation Neural networks Estimation Transforms Predictive models Prediction algorithms Iterative algorithms

来源：评论

学校读者我要写书评

暂无评论

Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity speech Synthesis

arXiv

引用

arXiv 2023年

作者： Lu, Ye-Xin Ai, Yang Ling, Zhen-Hua National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China

This paper proposes a source-filter-based generative adversarial neural vocoder named SF-GAN, which achieves high-fidelity waveform generation from input acoustic features by introducing F0-based source excitation signals to a neural filter framework. The SF-GAN vocoder is composed of a source module and a resolution-wise conditional filter module and is trained based on generative adversarial strategies. The source module produces an excitation signal from the F0 information, then the resolution-wise convolutional filter module combines the excitation signal with processed acoustic features at various temporal resolutions and finally reconstructs the raw waveform. The experimental results show that our proposed SF-GAN vocoder outperforms the state-of-the-art HiFi-GAN and Fre-GAN in both analysis-synthesis (AS) and text-to-speech (TTS) tasks, and the synthesized speech quality of SF-GAN is comparable to the ground-truth audio. Copyright © 2023, The Authors. All rights reserved.

关键词： Generative adversarial networks

来源：评论

学校读者我要写书评

暂无评论

ZERO-SHOT PERSONALIZED LIP-TO-speech SYNTHESIS WITH FACE IMAGE BASED VOICE CONTROL

arXiv

引用

arXiv 2023年

作者： Sheng, Zheng-Yan Ai, Yang Ling, Zhen-Hua National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China

Lip-to-speech (Lip2speech) synthesis, which predicts corresponding speech from talking face images, has witnessed significant progress with various models and training strategies in a series of independent studies. However, existing studies can not achieve voice control under zero-shot condition, because extra speaker embeddings need to be extracted from natural reference speech and are unavailable when only the silent video of an unseen speaker is given. In this paper, we propose a zero-shot personalized Lip2speech synthesis method, in which face images control speaker identities. A variational autoencoder is adopted to disentangle the speaker identity and linguistic content representations, which enables speaker embeddings to control the voice characteristics of synthetic speech for unseen speakers. Furthermore, we propose associated cross-modal representation learning to promote the ability of face-based speaker embeddings (FSE) on voice control. Extensive experiments verify the effectiveness of the proposed method whose synthetic utterances are more natural and matching with the personality of input video than the compared methods. To our best knowledge, this paper makes the first attempt on zero-shot personalized Lip2speech synthesis with a face image rather than reference audio to control voice characteristics. Copyright © 2023, The Authors. All rights reserved.

关键词： speech synthesis

来源：评论

学校读者我要写书评

暂无评论

DIFFUSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion

arXiv

引用

arXiv 2023年

作者： Tan, Chao-Hong Gu, Jia-Chen Ling, Zhen-Hua National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China

Diffusion models have emerged as the new state-of-the-art family of deep generative models, and their promising potentials for text generation have recently attracted increasing attention. Existing studies mostly adopt a single encoder architecture with partially noising processes for conditional text generation, but its degree of flexibility for conditional modeling is limited. In fact, the encoder-decoder architecture is naturally more flexible for its detachable encoder and decoder modules, which is extensible to multilingual and multimodal generation tasks for conditions and target texts. However, the encoding process of conditional texts lacks the understanding of target texts. To this end, a spiral interaction architecture for encoder-decoder text diffusion (DiffuSIA) is proposed. Concretely, the conditional information from encoder is designed to be captured by the diffusion decoder, while the target information from decoder is designed to be captured by the conditional encoder. These two types of information flow run through multi-layer interaction spirally for deep fusion and understanding. DiffuSIA is evaluated on four text generation tasks, including paraphrase, text simplification, question generation, and open-domain dialogue generation. Experimental results show that DiffuSIA achieves competitive performance among previous methods on all four tasks, demonstrating the effectiveness and generalization ability of the proposed method.1 Copyright © 2023, The Authors. All rights reserved.

关键词： Diffusion

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：