检索结果-内蒙古大学图书馆

arXiv 2023年

作者： Wang, Guanqun Liu, Jiaming Li, Chenxuan Ma, Junpeng Zhang, Yuan Wei, Xinyu Zhang, Kevin Chong, Maurice Zhang, Ray Liu, Yijiang Zhang, Shanghang National Key Laboratory for Multimedia Information Processing School of Computer Science Peking University China Shanghai AI Lab China Nanjing University China

The burgeoning field of Multimodal Large Language Models (MLLMs) has exhibited remarkable performance in diverse tasks such as captioning, commonsense reasoning, and visual scene understanding. However, the deployment of these large-scale MLLMs on client devices is hindered by their extensive model parameters, leading to a notable decline in generalization capabilities when these models are compressed for device deployment. Addressing this challenge, we introduce a Cloud-Device Collaborative Continual Adaptation framework, designed to enhance the performance of compressed, device-deployed MLLMs by leveraging the robust capabilities of cloud-based, larger-scale MLLMs. Our framework is structured into three key components: a device-to-cloud uplink for efficient data transmission, cloud-based knowledge adaptation, and an optimized cloud-to-device downlink for model deployment. In the uplink phase, we employ an Uncertainty-guided Token Sampling (UTS) strategy to effectively filter out-of-distribution tokens, thereby reducing transmission costs and improving training efficiency. On the cloud side, we propose Adapter-based Knowledge Distillation (AKD) method to transfer refined knowledge from large-scale to compressed, pocket-size MLLMs. Furthermore, we propose a Dynamic Weight update Compression (DWC) strategy for the downlink, which adaptively selects and quantizes updated weight parameters, enhancing transmission efficiency and reducing the representational disparity between cloud and device models. Extensive experiments on several multimodal benchmarks demonstrate the superiority of our proposed framework over prior Knowledge Distillation and device-cloud collaboration methods. Notably, we also validate the feasibility of our approach to real-world experiments. Copyright © 2023, The Authors. All rights reserved.

关键词： Distillation

来源：评论

学校读者我要写书评

暂无评论

Rethink Video Retrieval Representation for Video Captioning

SSRN

引用

SSRN 2024年

作者： Tian, Mingkai Li, Guorong Qi, Yuankai Wang, Shuhui Sheng, Quan Z. Huang, Qingming School of Computer Science and Technology Key Lab of Big Data Mining and Knowledge Management University of Chinese Academy of Sciences China School of Computing Macquarie University Australia Key Laboratory of Intelligent Information Processing Institute of Computer Technology Chinese Academy of Sciences China

Video captioning, a challenging task targeting the automatic generation of accurate and comprehensive descriptions based on video content, has witnessed substantial success recently driven by bridging video representations and textual semantics. Inspired by the nature of the video retrieval task, which learns visual features strongly related to text queries, we propose to take advantage of visual representation learning from the video retrieval framework to tackle video captioning tasks. However, a simple direct application of video retrieval models performs poorly due to the weak ability to capture sufficient video details and temporal information required for video captioning. To increase the attention on details, we propose a learnable token shift module, which flexibly captures subtle movements in local regions across the temporal sequence. Furthermore, we devise a Refineformer, which learns to integrate local video tokens strongly related to desired captions via a cross-attention mechanism. Extensive experiments on MSVD, MSR-VTT and VATEX demonstrate the favorable performance of our method. © 2024, The Authors. All rights reserved.

关键词： Semantics

来源：评论

学校读者我要写书评

暂无评论

Semi-Supervised Self-Learning Enhanced Music Emotion Recognition

arXiv

引用

arXiv 2024年

作者： Sun, Yifu Zhang, Xulong Zhou, Monan Li, Wei School of Computer Science and Technology Fudan University Shanghai China Ping An Technology Co. Ltd. Shenzhen China Department of Music AI and Information Technology Central Conservatory of Music Beijing China Shanghai Key Laboratory of Intelligent Information Processing Fudan University Shanghai China

Music emotion recognition (MER) aims to identify the emotions conveyed in a given musical piece. However, currently, in the field of MER, the available public datasets have limited sample sizes. Recently, segment-based methods for emotion-related tasks have been proposed, which train backbone networks on shorter segments instead of entire audio clips, thereby naturally augmenting training samples without requiring additional resources. Then, the predicted segment-level results are aggregated to obtain the entire song prediction. The most commonly used method is that the segment inherits the label of the clip containing it, but music emotion is not constant during the whole clip. Doing so will introduce label noise and make the training easy to overfit. To handle the noisy label issue, we propose a semi-supervised self-learning (SSSL) method, which can differentiate between samples with correct and incorrect labels in a self-learning manner, thus effectively utilizing the augmented segment-level data. Experiments on three public emotional datasets demonstrate that the proposed method can achieve better or comparable performance. © 2024, CC BY.

关键词： Semi-supervised learning

来源：评论

学校读者我要写书评

暂无评论

Cross-Point Adversarial Attack Based on Feature Neighborhood Disruption Against Segment Anything Model

Cross-Point Adversarial Attack Based on Feature Neighborhood...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Yan Jiang Guisheng Yin Ye Yuan Jingjing Chen Zhipeng Wei College of Computer Science and Technology Harbin Engineering University Harbin China Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University Shanghai China Shanghai Collaborative Innovation Center of Intelligent Visual Computing Shanghai China

ISBN: (数字)9798350390155

ISBN: (纸本)9798350390162

Segment anything model (SAM) has received significant attention owing to its outstanding segmentation performance. However, it may still face security threats from adversarial examples. Since SAM interactively realizes the prediction of target areas according to user-specified prompts (e.g., points), adversarial examples generated by existing end-to-end attack methods usually exhibit limited attack performance when faced with different point prompts. To this end, we propose a cross-point adversarial attack method based on feature neighborhood disruption against SAM, called CP-FND attack. CP-FND aims to generate adversarial examples capable of effectively deceiving SAM under different user-specified point prompts. Specifically, CP-FND forces the intermediate feature of adversarial examples to be similar to the designed disruption features without relying on any specified point prompt. Subsequently, the continuity and relevance of contextual features are disrupted, thereby fooling SAM and suppressing its predicted masks. Extensive experiments demonstrate that CP-FND achieves superior cross-point adversarial attack performance against SAM compared to state-of-the-art methods.

关键词： Image segmentation Correlation Security Faces

来源：评论

学校读者我要写书评

暂无评论

General Compression Framework for Efficient Transformer Object Tracking

arXiv

引用

arXiv 2024年

作者： Hong, Lingyi Li, Jinglun Zhou, Xinyu Yan, Shilin Guo, Pinxue Jiang, Kaixun Chen, Zhaoyu Gao, Shuyong Zhang, Wei Lu, Hong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China

Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model’s ability to replicate the teacher model’s behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model’s compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17× speed up. © 2024, CC BY.

关键词： Students

来源：评论

学校读者我要写书评

暂无评论

Prototypical Residual Networks for Anomaly Detection and Localization

Prototypical Residual Networks for Anomaly Detection and Loc...

引用

Conference on computer Vision and Pattern Recognition (CVPR)

作者： Hui Zhang Zuxuan Wu Zheng Wang Zhineng Chen Yu-Gang Jiang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing School of Computer Science Zhejiang University of Technology

Anomaly detection and localization are widely used in industrial manufacturing for its efficiency and effectiveness. Anomalies are rare and hard to collect and supervised models easily over-fit to these seen anomalies with a handful of abnormal samples, producing unsatisfactory performance. On the other hand, anomalies are typically subtle, hard to discern, and of various appearance, making it difficult to detect anomalies and let alone locate anomalous regions. To address these issues, we propose a framework called Prototypical Residual Network (PRN), which learns feature residuals of varying scales and sizes between anomalous and normal patterns to accurately reconstruct the segmentation maps of anomalous regions. PRN mainly consists of two parts: multi-scale prototypes that explicitly represent the residual features of anomalies to normal patterns; a multisize self-attention mechanism that enables variable-sized anomalous feature learning. Besides, we present a variety of anomaly generation strategies that consider both seen and unseen appearance variance to enlarge and diversify anomalies. Extensive experiments on the challenging and widely used MVTec AD benchmark show that PRN outperforms current state-of-the-art unsupervised and supervised methods. We further report SOTA results on three additional datasets to demonstrate the effectiveness and generalizability of PRN.

关键词：

来源：评论

学校读者我要写书评

暂无评论

HiREN: Towards Higher Supervision Quality for Better Scene Text Image Super-Resolution

arXiv

引用

arXiv 2023年

作者： Zhao, Minyi Xu, Yi Li, Bingjia Wang, Jie Guan, Jihong Zhou, Shuigeng Shanghai Key Lab of Intelligent Information Processing School of Computer Science Jiangwan Campus Fudan University 2005 Songhu Road Shanghai200438 China ByteDance Inc Beijing100098 China Department of Computer Science and Technology Tongji University 4800 Caoan Road Shanghai201804 China

Scene text image super-resolution (STISR) is an important pre-processing technique for text recognition from low-resolution scene images. Nowadays, various methods have been proposed to extract text-specific information from high-resolution (HR) images to supervise STISR model training. However, due to uncontrollable factors (e.g. shooting equipment, focus, and environment) in manually photographing HR images, the quality of HR images cannot be guaranteed, which unavoidably impacts STISR performance. Observing the quality issue of HR images, in this paper we propose a novel idea to boost STISR by first enhancing the quality of HR images and then using the enhanced HR images as supervision to do STISR. Concretely, we develop a new STISR framework, called High-Resolution ENhancement (HiREN) that consists of two branches and a quality estimation module. The first branch is developed to recover the low-resolution (LR) images, and the other is an HR quality enhancement branch aiming at generating high-quality (HQ) text images based on the HR images to provide more accurate supervision to the LR images. As the degradation from HQ to HR may be diverse, and there is no pixel-level supervision for HQ image generation, we design a kernel-guided enhancement network to handle various degradation, and exploit the feedback from a recognizer and text-level annotations as weak supervision signal to train the HR enhancement branch. Then, a quality estimation module is employed to evaluate the qualities of HQ images, which are used to suppress the erroneous supervision information by weighting the loss of each image. Extensive experiments on TextZoom show that HiREN can work well with most existing STISR methods and significantly boost their performances. © 2023, CC BY-NC-SA.

关键词： Image enhancement

来源：评论

学校读者我要写书评

暂无评论

From Efficient Multimodal Models to World Models: A Survey

arXiv

引用

arXiv 2024年

作者： Mai, Xinji Tao, Zeng Lin, Junxiong Wang, Haoran Chang, Yang Kang, Yanlan Wang, Yan Zhang, Wenqiang Shanghai Engineering Research Center of AI and Robotics Academy for Engineering and Technology Fudan University Shanghai China Engineering Research Center of AI and Robotics Ministry of Education Academy for Engineering and Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field. Copyright © 2024, The Authors. All rights reserved.

关键词： Decision making

来源：评论

学校读者我要写书评

暂无评论

P3S-Diffusion:A Selective Subject-driven Generation Framework via Point Supervision

arXiv

引用

arXiv 2024年

作者： Hu, Junjie Gao, Shuyong Hong, Lingyi Wang, Qishan Zhao, Yuzhou Wang, Yan Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Recent research in subject-driven generation increasingly emphasizes the importance of selective subject features. Nevertheless, accurately selecting the content in a given reference image still poses challenges, especially when selecting the similar subjects in an image (e.g., two different dogs). Some methods attempt to use text prompts or pixel masks to isolate specific elements. However, text prompts often fall short in precisely describing specific content, and pixel masks are often expensive. To address this, we introduce P3S-Diffusion, a novel architecture designed for context-selected subject-driven generation via point supervision. P3S-Diffusion leverages minimal cost label (e.g., points) to generate subject-driven images. During fine-tuning, it can generate an expanded base mask from these points, obviating the need for additional segmentation models. The mask is employed for inpainting and aligning with subject representation. The P3S-Diffusion preserves fine features of the subjects through Multi-layers Condition Injection. Enhanced by the Attention Consistency Loss for improved training, extensive experiments demonstrate its excellent feature preservation and image generation capabilities. Copyright © 2024, The Authors. All rights reserved.

关键词： Pixels

来源：评论

学校读者我要写书评

暂无评论

Unveiling and Consulting Core Experts in Retrieval-Augmented MoE-based LLMs

arXiv

引用

arXiv 2024年

作者： Zhou, Xin Nie, Ping Guo, Yiwen Wei, Haojie Zhang, Zhanqiu Minervini, Pasquale Ma, Ruotian Gui, Tao Zhang, Qi Huang, Xuanjing School of Computer Science Fudan University Shanghai China LightSpeed Studios Tencent China Institute of Modern Languages and Linguistics Fudan University Shanghai China Key Laboratory of Intelligent Information Processing Fudan University Shanghai China School of Informatics and ELLIS University of Edinburgh United Kingdom

Retrieval-Augmented Generation (RAG) significantly improved the ability of Large Language Models (LLMs) to solve knowledge-intensive tasks. While existing research seeks to enhance RAG performance by retrieving higher-quality documents or designing RAG-specific LLMs, the internal mechanisms within LLMs that contribute to the effectiveness of RAG systems remain underexplored. In this paper, we aim to investigate these internal mechanisms within the popular Mixture-of-Expert (MoE)- based LLMs and demonstrate how to improve RAG by examining expert activations in these LLMs. Our controlled experiments reveal that several core groups of experts are primarily responsible for RAG-related behaviors. The activation of these core experts can signify the model's inclination towards external/internal knowledge and adjust its behavior. For instance, we identify core experts that can (1) indicate the sufficiency of the model's internal knowledge, (2) assess the quality of retrieved documents, and (3) enhance the model's ability to utilize context. Based on these findings, we propose several strategies to enhance RAG's efficiency and effectiveness through expert activation. Experimental results across various datasets and MoE-based LLMs show the effectiveness of our method. Copyright © 2024, The Authors. All rights reserved.

关键词： Natural language processing systems

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：