检索结果-内蒙古大学图书馆

UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Du, Chenpeng Guo, Yiwei Shen, Feiyu Liu, Zhijun Liang, Zheng Chen, Xie Wang, Shuai Zhang, Hui Yu, Kai MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Shenzhen Research Institute of Big Data Shenzhen China AISpeech Ltd Beijing China

The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTXvec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing. Copyright © 2023, The Authors. All rights reserved.

关键词： Semantics

Towards Effective and Compact Contextual Representation for Conformer Transducer Speech Recognition Systems

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Cui, Mingyu Kang, Jiawen Deng, Jiajun Yin, Xi Xie, Yutao Chen, Xie Liu, Xunying The Chinese University of Hong Kong Hong Kong MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Peng Cheng Laboratory Shenzhen China

Current ASR systems are mainly trained and evaluated at the utterance level. Long range cross utterance context can be incorporated. A key task is to derive a suitable compact representation of the most relevant history contexts. In contrast to previous researches based on either LSTM-RNN encoded histories that attenuate the information from longer range contexts, or frame level concatenation of transformer context embeddings, in this paper compact low-dimensional cross utterance contextual features are learned in the Conformer-Transducer Encoder using specially designed attention pooling layers that are applied over efficiently cached preceding utterances' history vectors. Experiments on the 1000-hr Gigaspeech corpus demonstrate that the proposed contextualized streaming Conformer-Transducers outperform the baseline using utterance internal context only with statistically significant WER reductions of 0.7% to 0.5% absolute (4.3% to 3.1% relative) on the dev and test data. Copyright © 2023, The Authors. All rights reserved.

关键词： Transducers

WebSRC: A Dataset for Web-Based Structural Reading Comprehension

学校读者我要写书评

暂无评论

WebSRC: A Dataset for Web-Based Structural Reading Comprehen...

2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021

作者： Chen, Xingyu Zhao, Zihan Chen, Lu Jiabao, J.I. Zhang, Danyang Luo, Ao Xiong, Yuxuan Yu, Kai X-LANCE Lab Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai Jiao Tong University Shanghai China State Key Lab of Media Convergence Production Technology and Systems Beijing China

ISBN: (纸本)9781955917094

Web search is an essential way for humans to obtain information, but it's still a great challenge for machines to understand the contents of web pages. In this paper, we introduce the task of structural reading comprehension (SRC) on web. Given a web page and a question about it, the task is to find the answer from the web page. This task requires a system not only to understand the semantics of texts but also the structure of the web page. Moreover, we proposed WebSRC, a novel Web-based Structural Reading Comprehension dataset. WebSRC consists of 400K question-answer pairs, which are collected from 6.4K web pages. Along with the QA pairs, corresponding HTML source code, screenshots, and metadata are also provided in our dataset. Each question in WebSRC requires a certain structural understanding of a web page to answer, and the answer is either a text span on the web page or yes/no. We evaluate various baselines on our dataset to show the difficulty of our task. We also investigate the usefulness of structural information and visual features. Our dataset and baselines have been publicly available. © 2021 Association for Computational Linguistics

关键词： Websites

Multi-turn Dialogue Comprehension from a Topic-aware Perspective

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Ma, Xinbei Xu, Yi Zhao, Hai Zhang, Zhuosheng The Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University China

Dialogue related Machine Reading Comprehension requires language models to effectively decoupling and modeling on multi-turn dialogue passages. As a dialogue development goes after the intentions of participants, its topic may not keep constant through the whole passage. Hence, it is non-trivial to detect and leverage the topic shift in dialogue modeling. Topic modeling, although has been widely studied in plain text, deserves far more utilization in dialogue reading comprehension. This paper proposes to model multi-turn dialogues from a topic-aware perspective. We start with a dialogue segmentation algorithm to split a dialogue passage into topic-concentrated fragments in an unsupervised way. Then we use these fragments as topic-aware language processing units in further dialogue comprehension. On one hand, the split segments indict specific topic rather than mixed intention, thus show convenient on in-domain topic detection and location. For this task, we design a clustering system with a self-training auto-encoder, and we build two constructed datasets for evaluation. On the other hand, the split segments are appropriate element of multi-turn dialogue response selection. For this purpose, we further present a novel model, Topic-Aware Dual-Attention Matching (TADAM) Network, which takes topic segments as processing element and matches response candidates with a dual cross-attention. Empirical studies on three public benchmarks show great improvements over baselines. Our work continues the previous studies on document topic, and brings the dialogue modeling to a novel topic-aware perspective with exhaustive experiments and analyses. Copyright © 2023, The Authors. All rights reserved.

关键词： Modeling languages

Evaluate Confidence Instead of Perplexity for Zero-shot Commonsense Reasoning

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Peng, Letian Li, Zuchao Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University China

Commonsense reasoning is an appealing topic in natural language processing (NLP) as it plays a fundamental role in supporting the human-like actions of NLP systems. With large-scale language models as the backbone, unsupervised pre-training on numerous corpora shows the potential to capture commonsense knowledge. Current pre-trained language model (PLM)-based reasoning follows the traditional practice using perplexity metric. However, commonsense reasoning is more than existing probability evaluation, which is biased by word frequency. This paper reconsiders the nature of commonsense reasoning and proposes a novel commonsense reasoning metric, Non-Replacement Confidence (NRC). In detail, it works on PLMs according to the Replaced Token Detection (RTD) pre-training objective in ELECTRA, in which the corruption detection objective reflects the confidence on contextual integrity that is more relevant to commonsense reasoning than existing probability. Our proposed novel method boosts zero-shot performance on two commonsense reasoning benchmark datasets and further seven commonsense question-answering datasets. Our analysis shows that pre-endowed commonsense knowledge, especially for RTD-based PLMs, is essential in downstream reasoning. Copyright © 2022, The Authors. All rights reserved.

关键词： Natural language processing systems

IBSEN: Director-Actor Agent Collaboration for Controllable and Interactive Drama Script Generation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Han, Senyu Chen, Lu Lin, Li-Min Xu, Zhengshan Yu, Kai X-LANCE Lab Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence SJTU AI Institute Shanghai Jiao Tong University Shanghai China Department of Cultural Industry Management School of Media and Communication Shanghai Jiao Tong University Shanghai China Suzhou Laboratory Suzhou China

Large language models have demonstrated their capabilities in storyline creation and human-like character role-playing. Current language model agents mainly focus on reasonable behaviors from the level of individuals, and their behaviors might be hard to constraint on the level of the whole storyline. In this paper we introduce IBSEN, a director-actor coordinate agent framework that generates drama scripts and makes the plot played by agents more controllable. The director agent writes plot outlines that the user desires to see, instructs the actor agents to role-play their characters, and reschedules the plot when human players participate in the scenario to ensure the plot is progressing towards the objective. To evaluate the framework, we create a novel drama plot that involves several actor agents and check the interactions between them under the instruction of the director agent. Evaluation results show that our framework could generate complete, diverse drama scripts from only a rough outline of plot objectives, meanwhile maintaining the characteristics of characters in the drama. Our codes and prompts are available at https://***/OpenDFM/ibsen. © 2024, CC BY.

关键词： Computational linguistics

Build a SRE Challenge System: Lessons from VoxSRC 2022 and CNSRC 2022

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Chen, Zhengyang Han, Bing Xiang, Xu Huang, Houjun Liu, Bei Qian, Yanmin MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Department of Computer Science and Engineering Shanghai Jiao Tong University China AISpeech Ltd Suzhou China

Many speaker recognition challenges have been held to assess the speaker verification system in the wild and probe the performance limit. Voxceleb Speaker Recognition Challenge (VoxSRC), based on the voxceleb, is the most popular. Besides, another challenge called CN-Celeb Speaker Recognition Challenge (CNSRC) is also held this year, which is based on the Chinese celebrity multi-genre dataset CN-Celeb. Last year, our team participated in both speaker verification closed tracks in CNSRC 2022 and VoxSRC 2022, and achieved the 1st place and 3rd place respectively. In most system reports, the authors usually only provide a description of their systems but lack an effective analysis of their methods. In this paper, we will outline how to build a strong speaker verification challenge system and give a detailed analysis of each method compared with some other popular technical means. Copyright © 2022, The Authors. All rights reserved.

关键词： Speech recognition

SJTU-aiSPEECH System for VoxCeleb Speaker Recognition Challenge 2022

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Chen, Zhengyang Han, Bing Xiang, Xu Huang, Houjun Liu, Bei Qian, Yanmin MoE Key Lab of Artificial Intelligence AI Institute Department of Computer Science and Engineering X-LANCE Lab Shanghai Jiao Tong University China AISpeech Ltd Suzhou China

This report describes the SJTU-aiSPEECH system for the Voxceleb Speaker Recognition Challenge 2022. For track1, we implemented two kinds of systems, the online system and the offline system. Different ResNet-based backbones and loss functions are explored. Our final fusion system achieved 3rd place in track1. For track3, we implemented statistic adaptation and jointly training based domain adaptation. In the jointly training based domain adaptation, we jointly trained the source and target domain dataset with different training objectives to do the domain adaptation. We explored two different training objectives for target domain data, self-supervised learning based angular proto-typical loss and semi-supervised learning based classification loss with estimated pseudo labels. Besides, we used the dynamic loss-gate and label correction (DLG-LC) strategy to improve the quality of pseudo labels when the target domain objective is a classification loss. Our final fusion system achieved 4th place (very close to 3rd place, relatively less than 1%) in track3. Copyright © 2022, The Authors. All rights reserved.

关键词： Speech recognition

A Teleoperated Manipulator Control System Based on Surface Electromyography Using Deep Learning 12

学校读者我要写书评

暂无评论

A Teleoperated Manipulator Control System Based on Surface E...

12th International Conference on CYBER Technology in Automation, Control, and Intelligent Systems, CYBER 2022

作者： Lai, Zhiping Zhang, Xueze Wang, Junkongshuai Mu, Wei Wang, aiping Niu, Lan Zhang, Lihua Wang, Hongbo Kang, Xiaoyang Bin, Jianxiong Shanghai Engineering Research Center of Ai and Robotics Moe Frontiers Center for Brain Science Institute of Ai and Robotics Academy for Engineering and Technology Fudan University Laboratory for Neural Interface and Brain Computer Interface State Key Laboratory of Medical Neurobiology Engineering Research Center of Ai and Robotics Ministry of Education Shanghai200433 China Ji Hua Laboratory Guangdong Province Foshan528200 China Yiwu Research Institute of Fudan University Zhejiang Yiwu City322000 China Research Center for Intelligent Sensing Zhejiang Lab Hangzhou311121 China

ISBN: (数字)9781665472678

ISBN: (纸本)9781665472678

Gesture recognition based on surface electromyography (sEMG) is a popular topic in teleoperation field, which has the characteristics of convenience, maneuverability, flexibility and adaptability. With the continuous development of deep learning technology, the variety and accuracy of gesture recognition are constantly improving. In our previous research, a new neural network based on graph convolutional network and temporal convolutional network (STCN-GR) was proposed and achieved good results in gesture recognition task. In this paper, we develop a teleoperated manipulator control system based on STCN-GR combined with majority voting. The experimental results show that the performance of proposed system can achieve relatively excellent performance in recognition accuracy and computation speed. In this paper, we choose a mechanism hand as a peripheral to test the performance of the STCN-GR based control system, and the results show that this system can be applied in teleoperation in feature work to provide convenience for human life. © 2022 IEEE.

关键词： Remote control