检索结果-内蒙古大学图书馆

VCEMO: Multi-Modal Emotion Recognition for Chinese Voiceprints

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Tang, Jinghua Zhang, Liyun Lu, Yu Ding, Dian Yang, Lanqing Chen, Yi-Chao Bian, Minjie Li, Xiaoshan Xue, Guangtao Shanghai Jiao Tong University Shanghai China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China Shanghai Voicecomm Information Technology Co. Ltd. Shanghai China Shanghai Data Group Co. Ltd Shanghai200011 China

Emotion recognition can enhance humanized machine responses to user commands, while voiceprint-based perception systems can be easily inte grated into commonly used devices like smartphones and stereos. Despite having the largest number of speakers, there's a noticeable absence of high-quality cor pus datasets for emotion recognition using Chinese voiceprints. Hence, this paper introduces the VCEMO dataset to address this deficiency. The proposed dataset is constructed from everyday conversations and comprises over 100 users and 7,747 textual samples. Furthermore, this paper proposes a multimodal-based model as a benchmark, which effectively fuses speech, text, and external knowledge using a co-attention structure. The system employs contrastive learn-ing- based regulation for the uneven distribution of the dataset and the diversity of emotional expressions. The experiments demonstrate the significant improve ment of the proposed model over SOTA on the VCEMO and IEMOCAP datasets. Code and dataset will be released for research. Copyright © 2024, The Authors. All rights reserved.

关键词： Emotion Recognition

Instruction-Driven Game Engine: A Poker Case Study

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wu, Hongqiu Liu, Xingyuan Wang, Yan Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China Tencent China

The Instruction-Driven Game Engine (IDGE) project aims to democratize game development by enabling a large language model (LLM) to follow free-form game descriptions and generate game-play processes. The IDGE allows users to create games simply by natural language instructions, which significantly lowers the barrier for game development. We approach the learning process for IDGEs as a Next State Prediction task, wherein the model autoregressively predicts the game states given player actions. The computation of game states must be precise;otherwise, slight errors could corrupt the game-play experience. This is challenging because of the gap between stability and diversity. To address this, we train the IDGE in a curriculum manner that progressively increases its exposure to complex scenarios. Our initial progress lies in developing an IDGE for Poker, which not only supports a wide range of poker variants but also allows for highly individualized new poker games through natural language inputs. This work lays the groundwork for future advancements in transforming how games are created and played. Copyright © 2024, The Authors. All rights reserved.

关键词： Game design

Adaptive Guidance for Local Training in Heterogeneous Federated Learning

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Jianqing Liu, Yang Hua, Yang Cao, Jian Yang, Qiang Shanghai Jiao Tong University China Tsinghua University China Shanghai Artificial Intelligence Laboratory China Queen’s University Belfast United Kingdom Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China Hong Kong University of Science and Technology Hong Kong

Model heterogeneity poses a significant challenge in Heterogeneous Federated Learning (HtFL). In scenarios with diverse model architectures, directly aggregating model parameters is impractical, leading HtFL methods to incorporate an extra objective alongside the original local objective on each client to facilitate collaboration. However, this often results in a mismatch between the extra and local objectives. To resolve this, we propose Federated Learning-to-Guide (FedL2G1), a method that adaptively learns to guide local training in a federated manner, ensuring the added objective aligns with each client’s original goal. With theoretical guarantees, FedL2G utilizes only first-order derivatives w.r.t. model parameters, achieving a non-convex convergence rate of O(1/T). We conduct extensive experiments across two data heterogeneity and six model heterogeneity settings, using 14 heterogeneous model architectures (e.g., CNNs and ViTs). The results show that FedL2G significantly outperforms seven state-of-the-art methods. © 2024, CC BY-NC-ND.

关键词： Federated learning

CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Xinbei Zhang, Zhuosheng Zhao, Hai School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University China Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China

Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments, especially for graphical user interface (GUI) automation. However, those GUI agents require comprehensive cognition ability including exhaustive perception and reliable action response. We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP), to systematically improve the GUI automation performance. First, CEP facilitates the GUI perception through different aspects and granularity, including screenshots and complementary detailed layouts for the visual channel and historical actions for the textual channel. Second, CAP decomposes the action prediction into sub-problems: action type prediction and action target conditioned on the action type. With our technical design, our agent achieves new state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios. Code is available at https://***/xbmxb/CoCo-Agent. Copyright © 2024, The Authors. All rights reserved.

关键词： Graphical user interfaces

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yang, Dongjie Han, Xiaodong Gao, Yan Hu, Yao Zhang, Shilin Zhao, Hai Shanghai Jiao Tong University China Xiaohongshu Inc. China South China University of Technology China The Department of Computer Science and Engineering Shanghai Jiao Tong University Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China

Large Language Models (LLMs) have shown remarkable comprehension abilities but face challenges in GPU memory usage during inference, hindering their scalability for real-time applications like chatbots. To accelerate inference, we store computed keys and values (KV cache) in the GPU memory. Existing methods study the KV cache compression to reduce memory by pruning the pre-computed KV cache. However, they neglect the inter-layer dependency between layers and huge memory consumption in pre-computation. To explore these deficiencies, we find that the number of crucial keys and values that influence future generations decreases layer by layer and we can extract them by the consistency in attention weights. Based on the findings, we propose PyramidInfer, a method that compresses the KV cache by layer-wise retaining crucial context. PyramidInfer saves significant memory by computing fewer keys and values without sacrificing performance. Experimental results show PyramidInfer improves 2.2x throughput compared to Accelerate with over 54% GPU memory reduction in KV cache. Our code is available in https://***/mutonix/ pyramidinfer. © 2024, CC BY.

关键词： Cache memory

On the Robustness of Editing Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Xinbei Ju, Tianjie Qiu, Jiyang Zhang, Zhuosheng Zhao, Hai Liu, Lifeng Wang, Yulong School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University China Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China Baichuan Intelligent Technology China

Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. Model editing enables the manipulation of specific knowledge memories and the behavior of language generation without retraining. However, the robustness of model editing remains an open question. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI. We focus on three key research questions. RQ1: Can edited LLMs behave consistently resembling communicative AI in realistic situations? RQ2: To what extent does the rephrasing of prompts lead LLMs to deviate from the edited knowledge memory? RQ3: Which knowledge features are correlated with the performance and robustness of editing? Our empirical studies uncover a substantial disparity between existing editing methods and the practical application of LLMs. On rephrased prompts that are flexible but common in realistic applications, the performance of editing experiences a significant decline. Further analysis shows that more popular knowledge is memorized better, easier to recall, and more challenging to edit effectively. Copyright © 2024, The Authors. All rights reserved.

关键词： Computational linguistics

GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yao, Yao Li, Zuchao Zhao, Hai Department of Computer Science and Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China National Engineering Research Center for Multimedia Software School of Computer Science Wuhan University Wuhan430072 China

The burgeoning size of Large Language Models (LLMs) has led to enhanced capabilities in generating responses, albeit at the expense of increased inference times and elevated resource demands. Existing methods of acceleration, predominantly hinged on knowledge distillation, generally necessitate fine-tuning of considerably large models, posing a challenge for average users. Furthermore, present techniques for expediting inference and reducing costs operate independently. To address these issues, we introduce a novel and intuitive Guidance-based Knowledge Transfer (GKT) framework. This approach leverages a larger LLM as a "teacher" to create guidance prompts, paired with a smaller "student" model to finalize responses. Remarkably, GKT requires no fine-tuning and doesn’t necessitate the teacher and student models to have the same vocabulary, allowing for extensive batch generation to accelerate the process while ensuring user customization. GKT can be seamlessly integrated into cloud-edge collaboration architectures, and is versatile enough for plug-and-play application across various models. When utilizing ChatGPT as teacher model and Llama2-70B as the student model, we can achieve 95.00% of ChatGPT’s performance at 52% of the cost. The results highlight substantial enhancements in accuracy and processing speed on the GSM8K and CSQA datasets, surpassing the performance of using either the student or teacher models in isolation. Our code is publicly available at https://***/Zoeyyao27/GKT © 2024, CC BY.

关键词： Distillation

GLaPE: Gold Label-agnostic Prompt Evaluation and Optimization for Large Language Model

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Xuanchang Zhang, Zhuosheng Zhao, Hai College of Zhiyuan Shanghai Jiao Tong University China School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University China Department of Computer Science and Engineering Shanghai Jiao Tong University China Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering Shanghai Jiao Tong University China Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3 China

Despite the rapid progress of large language models (LLMs), their task performance remains sensitive to prompt design. Recent studies have explored leveraging the LLM itself as an optimizer to identify optimal prompts that maximize task accuracy. However, when evaluating prompts, such approaches heavily rely on elusive manually annotated gold labels to calculate task accuracy for each candidate prompt, which hinders its generality. To overcome the limitation, this work proposes GLaPE, a gold label-agnostic prompt evaluation method to alleviate dependence on gold labels. GLaPE is composed of two critical aspects: self-consistency evaluation of a single prompt and mutual-consistency refinement across multiple prompts. Experimental results on 8 widely-recognized reasoning tasks demonstrate that GLaPE can produce more effective prompts, achieving performance comparable to those derived from manually annotated gold labels. Analysis shows that GLaPE provides reliable evaluations aligned with accuracy, even in the absence of gold labels. Code is publicly available at https://***/thunderous77/GLaPE. Copyright © 2024, The Authors. All rights reserved.

关键词： Gold