检索结果-内蒙古大学图书馆

Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Zhang, Zhuosheng Zhao, Hai Liu, Longxiang The Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University China China

Training machines to understand natural language and interact with humans is one of the major goals of artificial intelligence. Recent years have witnessed an evolution from matching networks to pre-trained language models (PrLMs). In contrast to the plain-text modeling as the focus of the PrLMs, dialogue texts involve multiple speakers and reflect special characteristics such as topic transitions and structure dep.ndencies between distant utterances. However, the related PrLM models commonly represent dialogues sequentially by processing the pairwise dialogue history as a whole. Thus the hierarchical information on either utterance interrelation or speaker roles coupled in such representations is not well addressed. In this work, we propose compositional learning for holistic interaction across the utterances beyond the sequential contextualization from PrLMs, in order to capture the utterance-aware and speaker-aware representations entailed in a dialogue history. We decouple the contextualized word representations by masking mechanisms in Transformer-based PrLM, making each word only focus on the words in current utterance, other utterances, and two speaker roles (i.e., utterances of sender and utterances of the receiver), respectively. In addition, we employ domain-adaptive training strategies to help the model adapt to the dialogue domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets, achieving new state-of-the-art performance over previous methods. Copyright © 2023, The Authors. All rights reserved.

关键词： Deep neural networks

Human Attention Based Movie Summarization: Dataset and Baseline Model

学校读者我要写书评

暂无评论

Human Attention Based Movie Summarization: Dataset and Basel...

2022 IEEE International Conference on Multimedia and Expo, ICME 2022

作者： Zhao, Defang Zhu, Dandan Min, Xiongkuo Yue, Jiaomin Zhang, Kaiwei Zhou, Qiangqiang Zhai, Guangtao Yang, Xiaokang CloudWalk Technology Shanghai China School of Computer Science and Technology Donghua University Shanghai China Institute of Image Communication and Network Engineering Shanghai Jiao Tong University Shanghai China School of Software Jiangxi Normal University Nanchang China Ai Institute Shanghai Jiao Tong University MOE Key Lab of Artificial Intelligence Shanghai China

ISBN: (数字)9781665485630

ISBN: (纸本)9781665485630

The movie summarization model can automatically edit a condensed and succinct version of the movie by selecting the keyframes. Previous works mainly resort to hand-crafted heuristics and most of them are unsupervised. Supervised movie summarization is a new research field and, there is currently no publicly suitable dataset available. Moreover, existing works only focus on the movies themselves while neglecting the audiences, who have the most say in which part of the movie is more attractive. To deal with the aforementioned limitations, we establish a human attention based movie summarization dataset Movie50. Specifically, we explore the human attention variations when watching videos and have the following findings: (1) The attention of humans is concentrated when watching keyframes. (2) The attention of humans is distracted when watching non-keyframes. Inspired by these findings, we collect the eye fixations of 20 participants when watching 50 movies and propose a novel human attention based annotation pipeline. In addition, we introduce A/V-MSNet, an audiovisual neural network that takes advantage of spatio-temporal visual and auditory information to better model human attention as well as exploit more plentiful information. Extensive experiments demonstrate the superiority of the proposed method. © 2022 IEEE.

关键词： Motion pictures

Modeling Dynamic User Preference via Dictionary Learning for Sequential Recommendation

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Chen, Chao Li, Dongsheng Yan, Junchi Yang, Xiaokang School of Electronic Information and Electrical Engineering MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai200240 China Microsoft Research Asia Shanghai China School of Computer Science Fudan University Shanghai201203 China Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai200240 China MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai200240 China

Capturing the dynamics in user preference is crucial to better predict user future behaviors because user preferences often drift over time. Many existing recommendation algorithms-including both shallow and deep ones-often model such dynamics indep.ndently, i.e., user static and dynamic preferences are not modeled under the same latent space, which makes it difficult to fuse them for recommendation. This paper considers the problem of embedding a user's sequential behavior into the latent space of user preferences, namely translating sequence to preference. To this end, we formulate the sequential recommendation task as a dictionary learning problem, which learns: 1) a shared dictionary matrix, each row of which represents a partial signal of user dynamic preferences shared across users;and 2) a posterior distribution estimator using a deep autoregressive model integrated with Gated Recurrent Unit (GRU), which can select related rows of the dictionary to represent a user's dynamic preferences conditioned on his/her past behaviors. Qualitative studies on the Netflix dataset demonstrate that the proposed method can capture the user preference drifts over time and quantitative studies on multiple real-world datasets demonstrate that the proposed method can achieve higher accuracy compared with state-of-the-art factorization and neural sequential recommendation methods. The code is available at https://***/cchao0116/S2PNM-TKDE2021. © 2022, CC BY.

关键词： Collaborative filtering

EVASION: Efficient KV CAche CompreSsion vIa PrOduct QuaNtization

学校读者我要写书评

暂无评论

EVASION: Efficient KV CAche CompreSsion vIa PrOduct QuaNtiza...

2025 Design, Automation and Test in Europe Conference, DATE 2025

作者： Wang, Zongwu Liu, Fangxin Xu, Peng Sun, Qingxiao Zhao, Junping Jiang, Li China University of Petroleum-Beijing SSSLab Dept. of Cst China Ant Group China Shanghai Jiao Tong University Department of Computer Science and Engineering China Shanghai Qi Zhi Institute China Ai Institute Shanghai Jiao Tong University MoE Key Lab of Artificial Intelligence China

ISBN: (纸本)9783982674100

Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128K or 1M tokens. This trend, however, presents significant challenges in inference speed and memory management. The primary bottleneck in long-context LLM inference is the quadratic computational complexity of attention mechanisms, causing substantial slowdowns as sequence length increases. KV cache mechanism alleviates this issue by storing pre-computed data, but introduces memory requirements that scale linearly with context length, hindering efficient LLM dep.oyment. Quantization emerges as a promising approach to address the widening gap between LLM size and memory capacity. However, traditional quantization schemes often yield suboptimal compression results for KV caches due to two key factors: i) On-the-fly quantization and de-quantization, causing significant performance overhead;ii) Prevalence of outliers in KV values, challenging low-bitwidth uniform quantization. To this end, we propose EVASION, a novel quantization framework achieving low-bitwidth KV cache through product quantization. First, we conduct a thorough analysis of KV cache distribution, revealing the limitations of existing quantization schemes. Second, we introduce a non-uniform quantization algorithm based on product quantization, which efficiently compresses data while preserving accuracy. Third, we develop a high-performance GPU inference framework for EVASION that leverages sparse computation and asynchronous quantization, significantly enhancing inference speed. Comprehensive evaluation results demonstrate that EVASION can achieve 4 bits quantization trivial perplexity and accuracy loss. © 2025 EDAA.

关键词： Cache memory

Mixed Chain-of-Psychotherapies for Emotional Support Chatbot

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, Siyuan Ming, Cong Zhang, Zhiling Chen, Yanyi Zhu, Kenny Q. Wu, Mengyue X-LANCE Lab Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China Shanghai Jiao Tong University Shanghai China Rutgers University New Jersey United States University of Texas at Arlington ArlingtonTX United States

In the realm of mental health support chatbots, it is vital to show empathy and encourage self-exploration to provide tailored solutions. However, current approaches tend to provide general insights or solutions without fully understanding the help-seeker’s situation. Therefore, we propose PsyMix, a chatbot that integrates the analyses of the seeker’s state from the perspective of a psychotherapy approach (Chain-of-Psychotherapies, CoP) before generating the response, and learns to incorporate the strength of various psychotherapies by finetuning on a mixture of CoPs. Through comprehensive evaluation, we found that PsyMix can outperform the ChatGPT baseline, and demonstrate a comparable level of empathy in its responses to that of human counselors. Copyright © 2024, The Authors. All rights reserved.

关键词：

Speaking style compensation on synthetic audio for robust keyword spotting

学校读者我要写书评

暂无评论

Speaking style compensation on synthetic audio for robust ke...

International Symposium on Chinese Spoken Language Processing

作者： Houjun Huang BYanmin Qian AISpeech Ltd Suzhou China Department of Computer Science and Engineering MoE Key Lab of Artificial IntelligenceAI Institute X-LANCE Lab Shanghai Jiao Tong University Shanghai China

ISBN: (纸本)9798350397970

With the rise of intelligent speech processing applications, quickly producing keyword spotting (KWS) models with low resource has gained particular importance in recent years. Multi-speaker text-to-speech (TTS) has been proved to be an effective data augmentation technique for KWS to help complement inadequacies in the training data. However, in previous works, KWS system built with TTS augmented data couldn’t obtain considerable performance of that trained with real recordings as synthetic speeches could not fully represent target speaker’s speaking style. This work focuses on how to make synthesized speeches to be more similar to reference speaker’s speaking style under a specific metric. Speaker classification accuracy of synthesized keyword data on a speaker recognition model trained with real common recordings is used as the objective metrics, and the deep complex convolution recurrent network (DCCRN) is used to optimize it. Experimental results show that TTS augmentation helps improve the KWS system’s robustness. Moreover, by compensating speaking style of the synthetic data, we achieve a significant further improvement.

关键词： Measurement Training Convolution Training data Data models Speaker recognition Recording

META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Sun, Liangtai Chen, Xingyu Chen, Lu Dai, Tianle Zhu, Zichen Yu, Kai X-LANCE Lab Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai Jiao Tong University Shanghai China

Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current TOD systems usually focus on multi-turn text/speech interaction, then they would call back-end APIs designed for TODs to perform the task. However, this API-based architecture greatly limits the information-searching capability of intelligent assistants and may even lead to task failure if TOD-specific APIs are not available or the task is too complicated to be executed by the provided APIs. In this paper, we propose a new TOD architecture: GUI-based task-oriented dialogue system (GUI-TOD). A GUI-TOD system can directly perform GUI operations on real APPs and execute tasks without invoking TOD-specific backend APIs. Furthermore, we release META-GUI, a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. We also propose a multi-model action prediction and response model, which show promising results on META-GUI. The dataset, codes and leaderboard are publicly available © 2022, CC BY.

关键词： Graphical user interfaces

Improving Speech Separation with Knowledge Distilled from Self-supervised Pre-trained Models

学校读者我要写书评

暂无评论

Improving Speech Separation with Knowledge Distilled from Se...

International Symposium on Chinese Spoken Language Processing

作者： Bowen Qu Chenda Li Jinfeng Bai Yanmin Qian Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Shanghai Jiao Tong University Shanghai China TAL Education Group China

ISBN: (纸本)9798350397970

Large-scale self-supervised learning (SSL) models have shown outstanding ability in many speech processing tasks. Most of the SSL models in the literature are trained with datasets where the single-talker utterances dominate. It may not be optimal to directly apply these SSL models to speech separation tasks. Besides, many computational costs in large-scale SSL models increase the overall complexity of the speech separation system. In this paper, we explore the application of pre-trained SSL models in the speech separation task. Instead of using the SSL model directly, we designed an SSL feature predictor to estimate single-talker’s deep features from the speech mixture. The SSL feature predictor is trained with the knowledge distilled from the pre-trained Wav2Vec2.0 model. Our experiments show that the performance of time-domain speech separation can be improved obviously by leveraging the SSL embedding predictor.

关键词： Computational modeling Self-supervised learning Predictive models Feature extraction Computational efficiency Complexity theory Task analysis

DISENTANGLING THE PROSODY AND SEMANTIC INFORMATION WITH PRE-TRaiNED MODEL FOR IN-CONTEXT LEARNING BASED ZERO-SHOT VOICE CONVERSION

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, Zhengyang Wang, Shuai Zhang, Mingyang Liu, Xuechen Yamagishi, Junichi Qian, Yanmin Auditory Cognition and Computational Acoustics Lab MoE Key Lab of Artificial Intelligence AI Institute Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Shenzhen Research Institute of Big Data Shenzhen China Chinese University of Hong Kong Shenzhen China National Institute of Informatics Tokyo Japan

Voice conversion (VC) aims to modify the speaker's timbre while retaining speech content. Previous approaches have tokenized the outputs from self-supervised into semantic tokens, facilitating disentanglement of speech content information. Recently, in-context learning (ICL) has emerged in text-to-speech (TTS) systems for effectively modeling specific characteristics such as timbre through context conditioning. This paper proposes an ICL capability enhanced VC system (ICL-VC) employing a mask and reconstruction training strategy based on flow-matching generative models. Augmented with semantic tokens, our experiments on the LibriTTS dataset demonstrate that ICL-VC improves speaker similarity. Additionally, we find that k-means is a versatile tokenization method applicable to various pre-trained models. However, the ICL-VC system faces challenges in preserving the prosody of the source speech. To mitigate this issue, we propose incorporating prosody embeddings extracted from a pre-trained emotion recognition model into our system. Integration of prosody embeddings notably enhances the system's capability to preserve source speech prosody, as validated on the Emotional Speech Database. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics