检索结果-内蒙古大学图书馆

Leveraging Prompt Learning and Pause Encoding for Alzheimer’s Disease Detection

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Liu, Yin-Long Feng, Rui Yuan, Jia-Hong Ling, Zhen-Hua National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China Interdisciplinary Research Center for Linguistic Sciences University of Science and Technology of China Hefei China

Compared to other clinical screening techniques, speech-and-language-based automated Alzheimer’s disease (AD) detection methods are characterized by their non-invasiveness, cost-effectiveness, and convenience. Previous studies have demonstrated the efficacy of fine-tuning pre-trained language models (PLMs) for AD detection. However, the objective of this traditional fine-tuning method, which involves inputting only transcripts, is inconsistent with the masked language modeling (MLM) task used during the pre-training phase of PLMs. In this paper, we investigate prompt-based fine-tuning of PLMs, converting the classification task into a MLM task by inserting prompt templates into the transcript inputs. We also explore the impact of incorporating pause information from forced alignment into manual transcripts. Additionally, we compare the performance of various automatic speech recognition (ASR) models and select the Whisper model to generate ASR-based transcripts for comparison with manual transcripts. Furthermore, majority voting and ensemble techniques are applied across different PLMs (BERT and RoBERTa) using different random seeds. Ultimately, we obtain maximum detection accuracy of 95.8% (with mean 87.9%, std 3.3%) using manual transcripts, achieving state-of-the-art performance for AD detection using only transcripts on the ADReSS test set. Copyright © 2024, The Authors. All rights reserved.

关键词： Neurodegenerative diseases

Stargan-vc Based Cross-Domain Data Augmentation for Speaker Verification

学校读者我要写书评

暂无评论

Stargan-vc Based Cross-Domain Data Augmentation for Speaker ...

International Conference on Acoustics, speech, and Signal processing (ICASSP)

作者： Hang-Rui Hu Yan Song Jian-Tao Zhang Li-Rong Dai Ian McLoughlin Zhu Zhuo Yu Zhou Yu-Hong Li Hui Xue National Engineering Research Center for Speech and Language Information Processing University of Science and Technology of China Hefei China Alibaba Group China

Automatic speaker verification (ASV) faces domain shift caused by the mismatch of intrinsic and extrinsic factors, such as recording device and speaking style, in real-world applications, which leads to severe performance degradation. Since single-speaker multi-condition (SSMC) data is difficult to collect in practice, existing domain adaptation methods are hard to ensure the feature consistency of the same class but different domains. To this end, we propose a cross-domain data generation method to obtain a domain-invariant ASV system. Inspired by voice conversion (VC) task, a StarGAN based generative model first learns cross-domain mappings from SSMC data, and then generates missing domain data for all speakers, thus increasing the intra-class diversity of the training set. Considering the difference between ASV and VC task, we renovate the corresponding training objectives and network structure to make the adaptation task-specific. Evaluations on achieve a relative performance improvement of about 5-8% over the baseline in terms of minDCF and EER, outperforming the CNSRC winner’s system of the equivalent scale.

关键词： Training Performance evaluation Degradation Adaptation models Data models Acoustics Recording

MAT-SED: A Masked Audio Transformer with Masked-Reconstruction Based Pre-training for Sound Event Detection

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cai, Pengfei Song, Yan Li, Kang Song, Haoyu McLoughlin, Ian National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China China ICT Cluster Singapore Institute of Technology Singapore The Australian National University Australia

Sound event detection (SED) methods that leverage a large pre-trained Transformer encoder network have shown promising performance in recent DCASE challenges. However, they still rely on an RNN-based context network to model temporal dependencies, largely due to the scarcity of labeled data. In this work, we propose a pure Transformer-based SED model with masked-reconstruction based pre-training, termed MAT-SED. Specifically, a Transformer with relative positional encoding is first designed as the context network, pre-trained by the masked-reconstruction task on all available target data in a self-supervised way. Both the encoder and the context network are jointly fine-tuned in a semi-supervised manner. Furthermore, a global-local feature fusion strategy is proposed to enhance the localization capability. Evaluation of MAT-SED on DCASE2023 task4 surpasses state-of-the-art performance, achieving 0.587/0.896 PSDS1/PSDS2 respectively. © 2024, CC BY.

关键词： Self-supervised learning

Zero-Shot Personalized Lip-To-speech Synthesis with Face Image Based Voice Control

学校读者我要写书评

暂无评论

Zero-Shot Personalized Lip-To-Speech Synthesis with Face Ima...

International Conference on Acoustics, speech, and Signal processing (ICASSP)

作者： Zheng-Yan Sheng Yang Ai Zhen-Hua Ling National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei P.R. China

Lip-to-speech (Lip2speech) synthesis, which predicts corresponding speech from talking face images, has witnessed significant progress with various models and training strategies in a series of independent studies. However, existing studies can not achieve voice control under zero-shot condition, because extra speaker embeddings need to be extracted from natural reference speech and are unavailable when only the silent video of an unseen speaker is given. In this paper, we propose a zero-shot personalized Lip2speech synthesis method, in which face images control speaker identities. A variational autoencoder is adopted to disentangle the speaker identity and linguistic content representations, which enables speaker embeddings to control the voice characteristics of synthetic speech for unseen speakers. Furthermore, we propose associated cross-modal representation learning to promote the ability of face-based speaker embeddings (FSE) on voice control. Extensive experiments verify the effectiveness of the proposed method whose synthetic utterances are more natural and matching with the personality of input video than the compared methods. To our best knowledge, this paper makes the first attempt on zero-shot personalized Lip2speech synthesis with a face image rather than reference audio to control voice characteristics.

关键词： Representation learning Training Signal processing Predictive models Linguistics Acoustics speech synthesis

Neural speech Phase Prediction Based on Parallel Estimation Architecture and Anti-Wrapping Losses

学校读者我要写书评

暂无评论

Neural Speech Phase Prediction Based on Parallel Estimation ...

International Conference on Acoustics, speech, and Signal processing (ICASSP)

作者： Yang Ai Zhen-Hua Ling National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei P.R.China

This paper presents a novel speech phase prediction model which predicts wrapped phase spectra directly from amplitude spectra by neural networks. The proposed model is a cascade of a residual convolutional network and a parallel estimation architecture. The parallel estimation architecture is composed of two parallel linear convolutional layers and a phase calculation formula, imitating the process of calculating the phase spectra from the real and imaginary parts of complex spectra and strictly restricting the predicted phase values to the principal value interval. To avoid the error expansion issue caused by phase wrapping, we design anti-wrapping training losses defined between the predicted wrapped phase spectra and natural ones by activating the instantaneous phase error, group delay error and instantaneous angular frequency error using an anti-wrapping function. Experimental results show that our proposed neural speech phase prediction model outperforms the iterative Griffin-Lim algorithm and other neural network-based method, in terms of both reconstructed speech quality and generation speed.

关键词： Training Estimation Signal processing algorithms Predictive models Prediction algorithms Iterative algorithms Delays

Constraining Sequential Model Editing with Editing Anchor Compression

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Xu, Hao-Xiang Ma, Jun-Yu Ling, Zhen-Hua Zhang, Ningyu Gu, Jia-Chen National Engineering Research Center of Speech and Language Information Processing University of Science and Technology of China Hefei China Zhejiang University China University of California Los Angeles United States

Large language models (LLMs) struggle with hallucinations due to false or outdated knowledge. Given the high resource demands of retraining these models, there is an increasing focus on developing model editing. However, the general abilities of LLMs across downstream tasks are prone to significant degradation during sequential editing. This paper statistically observes that the parameter matrix after editing exhibits a significant deviation compared to its previous state as the number of edits increases. This serious deviation affects the original knowledge associations within LLMs and leads to the degradation of their general abilities. To this end, a framework termed Editing Anchor Compression (EAC) is proposed to constrain the deviation of the parameter matrix during sequential editing. It compresses the editing information by selecting editing anchors that are important in encoding new relations without deviating too much from the original matrix, thereby preserving the general abilities. Experiments of applying EAC to two popular editing methods on three LLMs across four tasks are conducted. Evaluation results show that EAC effectively minimizes unreasonable deviations caused by model editing, preserving over 70% of the general abilities while better retaining the editing knowledge compared to the original counterpart methods1 Copyright © 2025, The Authors. All rights reserved.

关键词： Matrix algebra

Non-Contrastive Self-Supervised Learning of Utterance-Level speech Representations

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Cho, Jaejin Pappagari, Raghavendra Żelasko, Piotr Moro-Velazquez, Laureano Villalba, Jesús Dehak, Najim Center for Language and Speech Processing Johns Hopkins University BaltimoreMD United States Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD United States

Considering the abundance of unlabeled speech data and the high labeling costs, unsupervised learning methods can be essential for better system development. One of the most successful methods is contrastive self-supervised methods, which require negative sampling: sampling alternative samples to contrast with the current sample (anchor). However, it is hard to ensure if all the negative samples belong to classes different from the anchor class without labels. This paper applies a non-contrastive self-supervised learning method on an unlabeled speech corpus to learn utterance-level embeddings. We used DIstillation with NO labels (DINO), proposed in computer vision, and adapted it to the speech domain. Unlike the contrastive methods, DINO does not require negative sampling. These embeddings were evaluated on speaker verification and emotion recognition. In speaker verification, the unsupervised DINO embedding with cosine scoring provided 4.38% EER on the VoxCeleb1 test trial. This outperforms the best contrastive self-supervised method by 40% relative in EER. An iterative pseudo-labeling training pipeline, not requiring speaker labels, further improved the EER to 1.89%. In emotion recognition, the DINO embedding performed 60.87, 79.21, and 56.98% in micro-f1 score on IEMOCAP, Crema-D, and MSP-Podcast, respectively. The results imply the generality of the DINO embedding to different speech applications. Copyright © 2022, The Authors. All rights reserved.

关键词： Emotion Recognition

Joint domain adaptation and speech bandwidth extension using time-domain GANs for speaker verification

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Kataria, Saurabh Villalba, Jesús Moro-Velázquez, Laureano Dehak, Najim Center for Language and Speech Processing Johns Hopkins University BaltimoreMD United States Human Language Technology Center of Excellence Johns Hopkins University BaltimoreMD United States

speech systems developed for a particular choice of acoustic domain and sampling frequency do not translate easily to others. The usual practice is to learn domain adaptation and bandwidth extension models independently. Contrary to this, we propose to learn both tasks together. Particularly, we learn to map narrowband conversational telephone speech to wideband microphone speech. We developed parallel and non-parallel learning solutions which utilize both paired and unpaired data. First, we first discuss joint and disjoint training of multiple generative models for our tasks. Then, we propose a two-stage learning solution where we use a pre-trained domain adaptation system for pre-processing in bandwidth extension training. We evaluated our schemes on a Speaker Verification downstream task. We used the JHU-MIT experimental setup for NIST SRE21, which comprises SRE16, SRE-CTS Superset and SRE21. Our results provide the first evidence that learning both tasks is better than learning just one. On SRE16, our best system achieves 22% relative improvement in Equal Error Rate w.r.t. a direct learning baseline and 8% w.r.t. a strong bandwidth expansion system. Copyright © 2022, The Authors. All rights reserved.

关键词： speech recognition

Exploring Prompt-based Multi-task Learning for Multimodal Dialog State Tracking and Immersive Multimodal Conversation 11

学校读者我要写书评

暂无评论

Exploring Prompt-based Multi-task Learning for Multimodal Di...

11th Dialog System technology Challenge, DSTC 2023

作者： Chen, Yirong Li, Ya Wang, Tao Xing, Xiaofen Xu, Xiangmin Liu, Quan Liu, Cong Hu, Guoping Guangdong Provincial Key Laboratory of Human Digital Twin School of EE South China University of Technology Guangzhou China iFLYTEK Research Hefei China Pazhou Lab. Guangzhou China School of Future Technology South China University of Technology Guangzhou China State Key Laboratory of Cognitive Intelligence Hefei China National Engineering Research Center of Speech and Language Information Processing Hefei China

With the rise of the metaverse, immersive multimodal conversation has attracted more and more researchers’ attention. Multimodal contexts will become more important for human-computer interaction in the metaverse, especially in shopping domain. Unlike traditional conversation tasks, immersive multimodal conversation has challenges such as multimodal ambiguous candidate identification and multimodal coreference resolution, which makes it more difficult to dialog state tracking and response generation, as described in SIMMC 2.1 challenge, a part of DSTC11. In particular, as the number of objects in the scene increases, the difficulty will increase dramatically. We proposed PMTLED (Prompt-based Multi-Task Learning Encoder-Decoder), in which different subtasks use different prompts to make the model tend to focus on the current subtask. We achieve the winner in ambiguous candidates indentification and runner-up in multimodal coreference resolution (MM-Coref), multimodal dialog state tracking (MM-DST) and assistant response generation. Our code and model are made publicly available at https://***/scutcyr/dstc11-simmc2.1-scut-bds-lab. © 2023 Association for Computational Linguistics.

关键词： human computer interaction