检索结果-内蒙古大学图书馆

Motion Guided Region Message Passing for Video Captioning

学校读者我要写书评

暂无评论

Motion Guided Region Message Passing for Video Captioning

International Conference on Computer Vision (ICCV)

作者： Shaoxiang Chen Yu-Gang Jiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai Collaborative Innovation Center on Intelligent Visual Computing

ISBN: (纸本)9781665428132

Video captioning is an important vision task and has been intensively studied in the computer vision community. Existing methods that utilize the fine-grained spatial information have achieved significant improvements, however, they either rely on costly external object detectors or do not sufficiently model the spatial/temporal relations. In this paper, we aim at designing a spatial information extraction and aggregation method for video captioning without the need of external object detectors. For this purpose, we propose a Recurrent Region Attention module to better extract diverse spatial features, and by employing Motion-Guided Cross-frame Message Passing, our model is aware of the temporal structure and able to establish high-order relations among the diverse regions across frames. They jointly encourage information communication and produce compact and powerful video representations. Furthermore, an Adjusted Temporal Graph Decoder is proposed to flexibly update video features and model high-order temporal relations during decoding. Experimental results on three benchmark datasets: MSVD, MSR-VTT, and VATEX demonstrate that our proposed method can outperform state-of-the-art methods.

关键词： Location awareness Computer vision Visualization Message passing Computational modeling Detectors Feature extraction

OUS: Bridging Scene Context and Facial Features to Overcome the Rigid Cognitive Problem 39

学校读者我要写书评

暂无评论

OUS: Bridging Scene Context and Facial Features to Overcome ...

39th Annual AAAI Conference on Artificial Intelligence, AAAI 2025

作者： Mai, Xinji Wang, Haoran Tao, Zeng Lin, Junxiong Yan, Shaoqi Wang, Yan Yu, Jiawen Tong, Xuan Li, Yating Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University China School of Electrical and Electronic Engineering Shanghai Insittute of Techonlogy China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China Engineering Research Center of AI & Robotics Ministry of Education Fudan University China

ISBN: (纸本)157735897X

Dynamic Facial Expression Recognition (DFER) is crucial for affective computing but often overlooks the impact of scene context. We have identified a significant issue in current DFER tasks: human annotators typically integrate emotions from various angles, including environmental cues and body language, whereas existing DFER methods tend to consider the scene as noise that needs to be filtered out, focusing solely on facial information. We refer to this as the Rigid Cognitive Problem. The Rigid Cognitive Problem can lead to discrepancies between the cognition of annotators and models in some samples. To align more closely with the human cognitive paradigm of emotions, we propose an Overall Understanding of the Scene DFER method (OUS). OUS effectively integrates scene and facial features, combining scene-specific emotional knowledge for DFER. Extensive experiments on the two largest datasets in the DFER field, DFEW and FERV39k, demonstrate that OUS significantly outperforms existing methods. By analyzing the Rigid Cognitive Problem, OUS successfully understands the complex relationship between scene context and emotional expression, closely aligning with human emotional understanding in real-world scenarios. Copyright © 2025, Association for the Advancement of Artificial Intelligence (***). All rights reserved.

关键词：

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Fu, Jiyuan Chen, Zhaoyu Jiang, Kaixun Guo, Haijing Wang, Jiafeng Gao, Shuyong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging modality interaction through embedding guidance and interaction enhancement. Specifically, attacking text at the embedding level while preserving semantics, as well as utilizing interaction image gradients to enhance constraints on perturbations of texts and images. Significantly, in the image-text retrieval task on Flickr30K dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL, CLIPViT and CLIPCNN by 8.11%-16.75% over state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior performance in cross-task generalization scenarios. Our work addresses the underexplored realm of transfer attacks on VLP models, shedding light on the importance of modality interaction for enhanced adversarial robustness. Copyright © 2024, The Authors. All rights reserved.

关键词： Generative adversarial networks

General Compression Framework for Efficient Transformer Object Tracking

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hong, Lingyi Li, Jinglun Zhou, Xinyu Yan, Shilin Guo, Pinxue Jiang, Kaixun Chen, Zhaoyu Gao, Shuyong Zhang, Wei Lu, Hong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China

Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model’s ability to replicate the teacher model’s behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model’s compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17× speed up. © 2024, CC BY.

关键词： Students

P3S-Diffusion:A Selective Subject-driven Generation Framework via Point Supervision

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hu, Junjie Gao, Shuyong Hong, Lingyi Wang, Qishan Zhao, Yuzhou Wang, Yan Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Recent research in subject-driven generation increasingly emphasizes the importance of selective subject features. Nevertheless, accurately selecting the content in a given reference image still poses challenges, especially when selecting the similar subjects in an image (e.g., two different dogs). Some methods attempt to use text prompts or pixel masks to isolate specific elements. However, text prompts often fall short in precisely describing specific content, and pixel masks are often expensive. To address this, we introduce P3S-Diffusion, a novel architecture designed for context-selected subject-driven generation via point supervision. P3S-Diffusion leverages minimal cost label (e.g., points) to generate subject-driven images. During fine-tuning, it can generate an expanded base mask from these points, obviating the need for additional segmentation models. The mask is employed for inpainting and aligning with subject representation. The P3S-Diffusion preserves fine features of the subjects through Multi-layers Condition Injection. Enhanced by the Attention Consistency Loss for improved training, extensive experiments demonstrate its excellent feature preservation and image generation capabilities. Copyright © 2024, The Authors. All rights reserved.

关键词： Pixels

From Efficient Multimodal Models to World Models: A Survey

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Mai, Xinji Tao, Zeng Lin, Junxiong Wang, Haoran Chang, Yang Kang, Yanlan Wang, Yan Zhang, Wenqiang Shanghai Engineering Research Center of AI and Robotics Academy for Engineering and Technology Fudan University Shanghai China Engineering Research Center of AI and Robotics Ministry of Education Academy for Engineering and Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field. Copyright © 2024, The Authors. All rights reserved.

关键词： Decision making

MotionEditor: Editing Video Motion via Content-Aware Diffusion

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Tu, Shuyuan Dai, Qi Cheng, Zhi-Qi Hu, Han Han, Xintong Wu, Zuxuan Jiang, Yu-Gang Shanghai Key Lab of Intell. Info. Processing School of CS Fudan University China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China Microsoft Research Asia China Carnegie Mellon University United States Huya Inc China

Existing diffusion-based video editing models have made gorgeous advances for editing attributes of a source video over time but struggle to manipulate the motion information while preserving the original protagonist’s appearance and background. To address this, we propose MotionEditor, a diffusion model for video motion editing. MotionEditor incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence. While ControlNet enables direct generation based on skeleton poses, it encounters challenges when modifying the source motion in the inverted noise due to contradictory signals between the noise (source) and the condition (reference). Our adapter complements ControlNet by involving source content to transfer adapted control signals seamlessly. Further, we build up a two-branch architecture (a reconstruction branch and an editing branch) with a high-fidelity attention injection mechanism facilitating branch interaction. This mechanism enables the editing branch to query the key and value from the reconstruction branch in a decoupled manner, making the editing branch retain the original background and protagonist appearance. We also propose a skeleton alignment algorithm to address the discrepancies in pose size and position. Experiments demonstrate the promising motion editing ability of MotionEditor, both qualitatively and quantitatively. Copyright © 2023, The Authors. All rights reserved.

关键词： Diffusion

Aligning Recommendation and Conversation via Dual Imitation

学校读者我要写书评

暂无评论

Aligning Recommendation and Conversation via Dual Imitation

2022 Conference on Empirical Methods in Natural Language processing, EMNLP 2022

作者： Zhou, Jinfeng Wang, Bo Huang, Minlie Zhao, Dongming Huang, Kun He, Ruifang Hou, Yuexian College of Intelligence and Computing Tianjin University Tianjin China State Key Laboratory of Communication Content Cognition People's Daily Online Beijing China The CoAI Group DCST Institute for Artificial Intelligence State Key Lab of Intelligent Technology and Systems China Beijing National Research Center for Information Science and Technology Tsinghua University Beijing100084 China AI Lab China Mobile Communication Group Tianjin Co. Ltd. China

Human conversations of recommendation naturally involve the shift of interests which can align the recommendation actions and conversation process to make accurate recommendations with rich explanations. However, existing conversational recommendation systems (CRS) ignore the advantage of user interest shift in connecting recommendation and conversation, which leads to an ineffective loose coupling structure of CRS. To address this issue, by modeling the recommendation actions as recommendation paths in a knowledge graph (KG), we propose DICR (Dual Imitation for Conversational Recommendation), which designs a dual imitation to explicitly align the recommendation paths and user interest shift paths in a recommendation module and a conversation module, respectively. By exchanging alignment signals, DICR achieves bidirectional promotion between recommendation and conversation modules and generates high-quality responses with accurate recommendations and coherent explanations. Experiments demonstrate that DICR outperforms the state-of-the-art models on recommendation and conversation performance with automatic, human, and novel explainability metrics. © 2022 Association for Computational Linguistics.

关键词： Knowledge graph

Hi-EF: Benchmarking Emotion Forecasting in Human-interaction

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Wang, Haoran Mai, Xinji Tao, Zeng Wang, Yan Yu, Jiawen Zhou, Ziheng Tong, Xuan Yan, Shaoqi Zhao, Qing Gao, Shuyong Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Engineering Research Center of Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China School of Information Science and Technology Fudan University Shanghai China

Affective Forecasting, a research direction in psychology that predicts individuals' future emotions, is often constrained by numerous external factors like social influence and temporal distance. To address this, we transform Affective Forecasting into a Deep Learning problem by designing an Emotion Forecasting paradigm based on two-party interactions. We propose a novel Emotion Forecasting (EF) task grounded in the theory that an individual's emotions are easily influenced by the emotions or other information conveyed during interactions with another person. To tackle this task, we have developed a specialized dataset, Humaninteraction- based Emotion Forecasting (Hi-EF), which contains 3069 two-party Multilayered-Contextual Interaction Samples (MCIS) with abundant affectiverelevant labels and three modalities. Hi-EF not only demonstrates the feasibility of the EF task but also highlights its potential. Additionally, we propose a methodology that establishes a foundational and referential baseline model for the EF task and extensive experiments are provided. The dataset and code is available HERE. Copyright © 2024, The Authors. All rights reserved.

关键词： Prediction models