RefXVC: Cross-Lingual Voice Conversion With Enhanced Reference Leveraging

作     者：Zhang, Mingyang Zhou, Yi Ren, Yi Zhang, Chen Yin, Xiang Li, Haizhou 

作者机构：Chinese Univ Hong Kong Shenzhen Res Inst Big Data Sch Data Sci Shenzhen 518172 Peoples R China Natl Univ Singapore Dept Elect & Comp Engn Singapore 119077 Singapore ByteDance AI Lab Speech & Audio Team Shanghai 201103 Peoples R China 

出 版 物：《IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING》 (IEEE ACM Trans. Audio Speech Lang. Process.)

年 卷 期：2024年第32卷

页      面：4146-4156页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 0702[理学-物理学] 

基　　金：Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen [B10120210117-KP02] Shenzhen Natural Science Foundation Key Project [JCYJ20220818103001002] National Natural Science Foundation of China Shenzhen Science and Technology Program [ZDSYS20230626091302006] 

主　　题：Timbre Task analysis Feature extraction Speech recognition Training Speech enhancement Data mining Cross-lingual voice conversion (XVC) multi-reference pitch normalization speaker embedding 

摘      要：This paper proposes RefXVC, a method for cross-lingual voice conversion (XVC) that leverages reference information to improve conversion performance. Previous XVC works generally take an average speaker embedding to condition the speaker identity, which does not account for the changing timbre of speech that occurs with different pronunciations. To address this, our method uses both global and local speaker embeddings to capture the timbre changes during speech conversion. Additionally, we observed a connection between timbre and pronunciation in different languages and utilized this by incorporating a timbre encoder and a pronunciation matching network into our model. Furthermore, we found that the variation in tones is not adequately reflected in a sentence, and therefore, we used multiple references to better capture the range of a speaker s voice. The proposed method outperformed existing systems in terms of both speech quality and speaker similarity, highlighting the effectiveness of leveraging reference information in cross-lingual voice conversion.