检索结果-内蒙古大学图书馆

Scale-Aware Task Message Transferring for Multi-Task Learning

学校读者我要写书评

暂无评论

Scale-Aware Task Message Transferring for Multi-Task Learnin...

IEEE International Conference on Multimedia and Expo (ICME)

作者： Shalayiding Sirejiding Yuxiang Lu Hongtao Lu Yue Ding Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China Department of Computer Science and Engineering MOE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Shanghai China

Exploring cross-task interaction has been the mainstream in recent multi-task learning for dense predictions. However, existing works that focus on excavating cross-task contextual information are briefly based on hierarchical all-scale features, ignoring the complementary information from task relationships in multi-scale features. Meanwhile, recent research advances rarely pay attention to efficient information transformation between tasks. In this paper, we propose a novel end-to-end multi-task learning network termed as Scale-Aware Task Message Transferring Network (SATMTN) to explore the task relationships in multi-scale features to enrich the contextual cross-task information in the all-scale features. Specifically, we design Multi-Scale Message Passing Decoder (MSMP) to model interaction information between tasks in the multi-scale features, which is characterized by stacked bi-directional fully connected graph neural networks. Further, we devise another All-Scale Message Passing decoder (ASMP) to extract rich contextual information between tasks in the all-scale features, which is based on adaptive message transformation in the graph. Extensive experiments are implemented to show that our method surpasses the current state-of-the-art works on public multi-task benchmarks.

关键词：

Neuronal Activation States as Sample Embeddings for Data Selection in Task-Specific Instruction Tuning

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Ma, Da Shang, Gonghu Chen, Zhi Qin, Libo Luo, Yijie Pan, Lei Fan, Shuai Chen, Lu Yu, Kai X-LANCE Lab Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence SJTU AI Institute Shanghai Jiao Tong University Shanghai China AISpeech Co. Ltd. Suzhou China ByteDance China School of Computer Science and Engineering Central South University China

Task-specific instruction tuning enhances the performance of large language models (LLMs) on specialized tasks, yet efficiently selecting relevant data for this purpose remains a challenge. Inspired by neural coactivation in the human brain, we propose a novel data selection method called NAS, which leverages neuronal activation states as embeddings for samples in the feature space. Extensive experiments show that NAS outperforms classical data selection methods in terms of both effectiveness and robustness across different models, datasets, and selection ratios. © 2025, CC BY.

关键词： Embeddings

A Simple Framework for Text-Supervised Semantic Segmentation

学校读者我要写书评

暂无评论

A Simple Framework for Text-Supervised Semantic Segmentation

Conference on computer Vision and Pattern Recognition (CVPR)

作者： Muyang Yi Quan Cui Hao Wu Cheng Yang Osamu Yoshie Hongtao Lu Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute Shanghai Jiao Tong University Waseda University ByteDance Inc.

Text-supervised semantic segmentation is a novel research topic that allows semantic segments to emerge with image-text contrasting. However, pioneering methods could be subject to specifically designed network architectures. This paper shows that a vanilla contrastive language-image pretraining (CLIP) model is an effective text-supervised semantic segmentor by itself. First, we reveal that a vanilla CLIP is inferior to localization and segmentation due to its optimization being driven by densely aligning visual and language representations. Second, we propose the locality-driven alignment (LoDA) to address the problem, where CLIP optimization is driven by sparsely aligning local representations. Third, we propose a simple segmentation (SimSeg) framework. LoDA and SimSeg jointly amelio-rate a vanilla CLIP to produce impressive semantic segmentation results. Our method outperforms previous state-of-the-art methods on PASCAL VOC 2012, PASCAL Context and COCO datasets by large margins. Code and models are available at ***/muyangyi/SimSeg.

关键词：

Target Sound Extraction with Variable Cross-Modality Clues

学校读者我要写书评

暂无评论

Target Sound Extraction with Variable Cross-Modality Clues

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Chenda Li Yao Qian Zhuo Chen Dongmei Wang Takuya Yoshioka Shujie Liu Yanmin Qian Michael Zeng Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Shanghai Jiao Tong University Microsoft Redmond WA USA

Automatic target sound extraction (TSE) is a machine learning approach to mimic the human auditory perception capability of attending to a sound source of interest from a mixture of sources. It often uses a model conditioned on a fixed form of target sound clues, such as a sound class label, which limits the ways in which users can interact with the model to specify the target sounds. To leverage variable number of clues cross modalities available in the inference phase, including a video, a sound event class, and a text caption, we propose a unified transformer-based TSE model architecture, where a multi-clue attention module integrates all the clues across the modalities. Since there is no off-the-shelf benchmark to evaluate our proposed approach, we build a dataset 1 based on public corpora, Audioset and AudioCaps. Experimental results for seen and unseen target-sound evaluation sets show that our proposed TSE model can effectively deal with a varying number of clues which improves the TSE performance and robustness against partially compromised clues.

关键词： Machine learning Signal processing Benchmark testing Transformers Robustness Acoustics Speech processing

ALIGNSUM: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preference

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Han, Yang Wang, Yiming Wang, Rui Chen, Lu Yu, Kai X-LANCE Lab Department of Computer Science and Engineering SJTU China MoE Key Lab of Artificial Intelligence SJTU AI Institute Shanghai Jiao Tong University Shanghai China Suzhou Laboratory Suzhou China

Text summarization tasks commonly employ Pre-trained Language Models (PLMs) to fit diverse standard datasets. While these PLMs excel in automatic evaluations, they frequently underperform in human evaluations, indicating a deviation between their generated summaries and human summarization preferences. This discrepancy is likely due to the low quality of fine-tuning datasets and the limited availability of high-quality human-annotated data that reflect true human preference. To address this challenge, we introduce a novel human summarization preference alignment framework ALIGNSUM. This framework consists of three parts: Firstly, we construct a Data Pymarid with extractive, abstractive, and human-annotated summary data. Secondly, we conduct the Gaussian Resampling to remove summaries with extreme lengths. Finally, we implement the two-stage hierarchical fine-tuning with Data Pymarid after Gaussian Resampling. We apply ALIGNSUM to PLMs on the human-annotated CNN/DailyMail and BBC XSum datasets. Experiments show that with ALIGNSUM, PLMs like BART-Large surpass 175B GPT-3 in both automatic and human evaluations. This demonstrates that ALIGNSUM significantly enhances the alignment of language models with human summarization preferences.1 © 2024, CC BY.

关键词： Spatio-temporal data

On the Effectiveness of Acoustic BPE in Decoder-Only TTS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Bohan Shen, Feiyu Guo, Yiwei Wang, Shuai Chen, Xie Yu, Kai MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Department of Computer Science and Engineering Shanghai Jiao Tong University China Shenzhen Research Institute of Big Data CUHK-Shenzhen China

Discretizing speech into tokens and generating them by a decoder-only model have been a promising direction for text-to-speech (TTS) and spoken language modeling (SLM). To shorten the sequence length of speech tokens, acoustic byte-pair encoding (BPE) has emerged in SLM that treats speech tokens from self-supervised semantic representations as characters to further compress the token sequence. But the gain in TTS has not been fully investigated, and the proper choice of acoustic BPE remains unclear. In this work, we conduct a comprehensive study on various settings of acoustic BPE to explore its effectiveness in decoder-only TTS models with semantic speech tokens. Experiments on LibriTTS verify that acoustic BPE uniformly increases the intelligibility and diversity of synthesized speech, while showing different features across BPE settings. Hence, acoustic BPE is a favorable tool for decoder-only TTS. © 2024, CC BY.

关键词： Semantics

A SURVEY ON SPEECH LARGE LANGUAGE MODELS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Peng, Jing Wang, Yucheng Xi, Yu Li, Xu Zhang, Xizhuo Yu, Kai MoE Key Lab of Artificial Intelligence AI Institute X-LANCE Lab Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China AISpeech Co. Ltd. Suzhou China

Large Language Models (LLMs) exhibit strong contextual understanding and remarkable multi-task performance. Therefore, researchers have been seeking to integrate LLMs in the broad sense of Spoken Language Understanding (SLU) field. Different from the traditional method of cascading LLMs to process text generated by Automatic Speech Recognition(ASR), new efforts have focused on designing architectures centered around Audio Feature Extraction - Multimodal Information Fusion - LLM Inference (Speech LLMs). This approach enables richer audio feature extraction while simultaneously facilitating end-to-end fusion of audio and text modalities, thereby achieving deeper understanding and reasoning from audio data. This paper elucidates the development of Speech LLMs, offering an in-dep.h analysis of system architectures and training strategies. Through extensive research and a series of targeted experiments, the paper assesses Speech LLMs’ advancements in Rich Audio Transcription and its potential for Cross-task Integration within the SLU field. Additionally, it indicates key challenges uncovered through experimentation, such as the Dormancy of LLMs under certain conditions. The paper further delves into the training strategies for Speech LLMs, proposing potential solutions based on these findings, and offering valuable insights and references for future research in this domain, as well as LLM applications in multimodal contexts. Copyright © 2024, The Authors. All rights reserved.

关键词： Information fusion

DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Shao, Hang Liu, Bei Wang, Wei Gong, Xun Qian, Yanmin Auditory Cognition and Computational Acoustics Lab MoE Key Lab of Artificial Intelligence AI Institute Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai China

As a popular multilingual and multitask pre-trained speech model, Whisper has the problem of curse of multilinguality. To enhance multilingual capabilities in small Whisper models, we propose DQ-Whisper, a novel joint distillation and quantization framework to compress Whisper for efficient inference. Firstly, we propose a novel dynamic matching distillation strategy. Then, a quantization-aware distillation framework is introduced to integrate quantization with distillation. Experimental results on various multilingual datasets show that our suggested distillation approach can effectively enhance the multilingual capabilities of small Whisper models without increasing computational costs. Up to 5.18x reduction in model size is achieved with marginal performance degradation. In addition, quantization is compatible with distillation, which can result in a higher compression rate. Copyright © 2023, The Authors. All rights reserved.

关键词： Speech recognition

ChemDFM-X: Towards Large Multimodal Model for Chemistry

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhao, Zihan Chen, Bo Li, Jingpiao Chen, Lu Wen, Liyang Wang, Pengyu Zhu, Zichen Zhang, Danyang Wan, Ziping Li, Yansi Dai, Zhongyang Chen, Xin Yu, Kai X-LANCE Lab Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence SJTU AI Institute Shanghai Jiao Tong University Shanghai200240 China Suzhou Laboratory Suzhou215123 China

Rapid developments of ai tools are expected to offer unprecedented assistance to the research of natural science including chemistry. However, neither existing unimodal task-specific specialist models nor emerging general large multimodal models (LMM) can cover the wide range of chemical data modality and task categories. To address the real demands of chemists, a cross-modal Chemical General Intelligence (CGI) system, which serves as a truly practical and useful research assistant utilizing the great potential of LMMs, is in great need. In this work, we introduce the first Cross-modal Dialogue Foundation Model for Chemistry (ChemDFM-X). Diverse multimodal data are generated from an initial modality by approximate calculations and task-specific model predictions. This strategy creates sufficient chemical training corpora, while significantly reducing excessive expense, resulting in an instruction-tuning dataset containing 7.6M data. After instruction finetuning, ChemDFM-X is evaluated on extensive experiments of different chemical tasks with various data modalities. The results demonstrate the capacity of ChemDFM-X for multimodal and inter-modal knowledge comprehension. ChemDFM-X marks a significant milestone toward aligning all modalities in chemistry, a step closer to CGI. Copyright © 2024, The Authors. All rights reserved.

关键词： Alkylation