检索结果-内蒙古大学图书馆

arXiv 2024年

作者： Hong, Lingyi Li, Jinglun Zhou, Xinyu Yan, Shilin Guo, Pinxue Jiang, Kaixun Chen, Zhaoyu Gao, Shuyong Zhang, Wei Lu, Hong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China

Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model’s ability to replicate the teacher model’s behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model’s compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17× speed up. © 2024, CC BY.

关键词： Students

来源：评论

学校读者我要写书评

暂无评论

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

arXiv

引用

arXiv 2024年

作者： Fu, Jiyuan Chen, Zhaoyu Jiang, Kaixun Guo, Haijing Wang, Jiafeng Gao, Shuyong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging modality interaction through embedding guidance and interaction enhancement. Specifically, attacking text at the embedding level while preserving semantics, as well as utilizing interaction image gradients to enhance constraints on perturbations of texts and images. Significantly, in the image-text retrieval task on Flickr30K dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL, CLIPViT and CLIPCNN by 8.11%-16.75% over state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior performance in cross-task generalization scenarios. Our work addresses the underexplored realm of transfer attacks on VLP models, shedding light on the importance of modality interaction for enhanced adversarial robustness. Copyright © 2024, The Authors. All rights reserved.

关键词： Generative adversarial networks

来源：评论

学校读者我要写书评

暂无评论

From Efficient Multimodal Models to World Models: A Survey

arXiv

引用

arXiv 2024年

作者： Mai, Xinji Tao, Zeng Lin, Junxiong Wang, Haoran Chang, Yang Kang, Yanlan Wang, Yan Zhang, Wenqiang Shanghai Engineering Research Center of AI and Robotics Academy for Engineering and Technology Fudan University Shanghai China Engineering Research Center of AI and Robotics Ministry of Education Academy for Engineering and Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field. Copyright © 2024, The Authors. All rights reserved.

关键词： Decision making

来源：评论

学校读者我要写书评

暂无评论

Flickr30K-CFQ: A Compact and Fragmented Query Dataset for Text-image Retrieval

arXiv

引用

arXiv 2024年

作者： Liu, Haoyu Song, Yaoxian Wang, Xuwu Zhu, Xiangru Li, Zhixu Song, Wei Li, Tiefeng Research Center for Intelligent Robotics Zhejiang Lab China Department of Engineering Mechanics Center for X-Mechanics Zhejiang University China Shanghai Key Laboratory of Data Science School of Computer Science Fudan University China

With the explosive growth of multi-modal information on the Internet, unimodal search cannot satisfy the requirement of Internet applications. Text-image retrieval research is needed to realize high-quality and efficient retrieval between different modalities. Existing text-image retrieval research is mostly based on general vision-language datasets (e.g. MS-COCO, Flickr30K), in which the query utterance is rigid and unnatural (i.e. verbosity and formality). To overcome the shortcoming, we construct a new Compact and Fragmented Query challenge dataset (named Flickr30K-CFQ) to model text-image retrieval task considering multiple query content and style, including compact and fine-grained entity-relation corpus. We propose a novel LLM-based Query-enhanced method using prompt engineering based on LLM. Experiments show that our proposed Flickr30-CFQ reveals the insufficiency of existing vision-language datasets in realistic text-image tasks. Our LLM-based Query-enhanced method applied on different existing text-image retrieval models improves query understanding performance both on public dataset and our challenge set Flickr30-CFQ with over 0.9% and 2.4% respectively. Our project can be available anonymously in https://***/view/Flickr30K-cfq. Copyright © 2024, The Authors. All rights reserved.

关键词： Image enhancement

来源：评论

学校读者我要写书评

暂无评论

D2SP: Dynamic Dual-Stage Purification Framework for Dual Noise Mitigation in Vision-based Affective Recognition.

arXiv

引用

arXiv 2024年

作者： Wang, Haoran Mai, Xinji Tao, Zeng Tong, Xuan Lin, Junxiong Wang, Yan Yu, Jiawen Wang, Boyang Yan, Shaoqi Zhao, Qing Zhou, Ziheng Gao, Shuyong Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China School of Information Science and Technology Fudan University Shanghai China Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

The contemporary state-of-the-art of Dynamic Facial Expression Recognition (DFER) technology facilitates remarkable progress by deriving emotional mappings of facial expressions from video content, underpinned by training on voluminous datasets. Yet, the DFER datasets encompass a substantial volume of noise data. Noise arises from low-quality captures that defy logical labeling, and instances that suffer from mislabeling due to annotation bias, engendering two principal types of uncertainty: the uncertainty regarding data usability and the uncertainty concerning label reliability. Addressing the two types of uncertainty, we have meticulously crafted a two-stage framework aiming at Seeking Certain data In extensive Uncertain data (SCIU). This initiative aims to purge the DFER datasets of these uncertainties, thereby ensuring that only clean, verified data is employed in training processes. To mitigate the issue of low-quality samples, we introduce the Coarse-Grained Pruning (CGP) stage, which assesses sample weights and prunes those deemed unusable due to their low weight. For samples with incorrect annotations, the Fine-Grained Correction (FGC) stage evaluates prediction stability to rectify mislabeled data. Moreover, SCIU is conceived as a universally compatible, plug-and-play framework, tailored to integrate seamlessly with prevailing DFER methodologies. Rigorous experiments across prevalent DFER datasets and against numerous benchmark methods substantiates SCIU’s capacity to markedly elevate performance metrics. Copyright © 2024, The Authors. All rights reserved.

关键词： Face recognition

来源：评论

学校读者我要写书评

暂无评论

LVOS: A Benchmark for Large-scale Long-term Video Object Segmentation

arXiv

引用

arXiv 2024年

作者： Hong, Lingyi Liu, Zhongying Chen, Wenchao Tan, Chenzhi Feng, Yuang Zhou, Xinyu Guo, Pinxue Li, Jinglun Chen, Zhaoyu Gao, Shuyong Zhang, Wei Zhang, Wenqiang Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science Fudan University Shanghai200433 China The Shanghai Engineering Research Center of AI&Robotics Academy for Engineering&Technology Fudan University Shanghai China Engineering Research Center of AI&Robotics Ministry of Education Academy for Engineering&Technology Fudan University Shanghai China The Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

Video object segmentation (VOS) aims to distinguish and track target objects in a video. Despite the excellent performance achieved by off-the-shell VOS models, existing VOS benchmarks mainly focus on short-term videos lasting about 5 seconds, where objects remain visible most of the time. However, these benchmarks poorly represent practical applications, and the absence of long-term datasets restricts further investigation of VOS in realistic scenarios. Thus, we propose a novel benchmark named LVOS, comprising 720 videos with 296,401 frames and 407,945 high-quality annotations. Videos in LVOS last 1.14 minutes on average, approximately 5 times longer than videos in existing datasets. Each video includes various attributes, especially challenges deriving from the wild, such as long-term reappearing and cross-temporal similar objects. Compared to previous benchmarks, our LVOS better reflects VOS models' performance in real scenarios. Based on LVOS, we evaluate 20 existing VOS models under 4 different settings and conduct a comprehensive analysis. On LVOS, these models suffer a large performance drop, highlighting the challenge of achieving precise tracking and segmentation in real-world scenarios. Attribute-based analysis indicates that key factor to accuracy decline is the increased video length, emphasizing LVOS's crucial role. We hope our LVOS can advance development of VOS in real scenes. Data and code are available at https://***/lvos. ***/. © 2024, CC BY.

关键词： Benchmarking

来源：评论

学校读者我要写书评

暂无评论

Improving Adversarial Transferability with Neighbourhood Gradient Information

arXiv

引用

arXiv 2024年

作者： Guo, Haijing Wang, Jiafeng Chen, Zhaoyu Jiang, Kaixun Hong, Lingyi Guo, Pinxue Li, Jinglun Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai200433 China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai200433 China Engineering Research Center of Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai200433 China

Deep neural networks (DNNs) are known to be susceptible to adversarial examples, leading to significant performance degradation. In black-box attack scenarios, a considerable attack performance gap between the surrogate model and the target model persists. This work focuses on enhancing the transferability of adversarial examples to narrow this performance gap. We observe that the gradient information around the clean image, i.e. Neighbourhood Gradient Information, can offer high transferability. Leveraging this, we propose the NGI-Attack, which incorporates Example Backtracking and Multiplex Mask strategies, to use this gradient information and enhance transferability fully. Specifically, we first adopt Example Backtracking to accumulate Neighbourhood Gradient Information as the initial momentum term. Multiplex Mask, which forms a multi-way attack strategy, aims to force the network to focus on non-discriminative regions, which can obtain richer gradient information during only a few iterations. Extensive experiments demonstrate that our approach significantly enhances the adversarial transferability. Especially, when attacking numerous defense models, we achieve an average attack success rate of 95.8%. Notably, our method can plugin with any off-the-shelf algorithm to improve their attack performance without additional time cost. Copyright © 2024, The Authors. All rights reserved.

关键词：

来源：评论

学校读者我要写书评

暂无评论

Suppressing Uncertainties in Degradation Estimation for Blind Super-Resolution

arXiv

引用

arXiv 2024年

作者： Lin, Junxiong Tao, Zen Tong, Xuan Mai, Xinji Wang, Haoran Wang, Boyang Wang, Yan Zhao, Qing Yu, Jiawen Lin, Yuxuan Yan, Shaoqi Gao, Shuyong Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China East China University of Science and Technology Shanghai China Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

The problem of blind image super-resolution aims to recover high-resolution (HR) images from low-resolution (LR) images with unknown degradation modes. Most existing methods model the image degradation process using blur kernels. However, this explicit modeling approach struggles to cover the complex and varied degradation processes encountered in the real world, such as high-order combinations of JPEG compression, blur, and noise. Implicit modeling for the degradation process can effectively overcome this issue, but a key challenge of implicit modeling is the lack of accurate ground truth labels for the degradation process to conduct supervised training. To overcome this limitations inherent in implicit modeling, we propose an Uncertainty-based degradation representation for blind Super-Resolution framework (USR). By suppressing the uncertainty of local degradation representations in images, USR facilitated self-supervised learning of degradation representations. The USR consists of two components: Adaptive Uncertainty-Aware Degradation Extraction (AUDE) and a feature extraction network composed of Variable Depth Dynamic Convolution (VDDC) blocks. To extract Uncertainty-based Degradation Representation from LR images, the AUDE utilizes the Self-supervised Uncertainty Contrast module with Uncertainty Suppression Loss to suppress the inherent model uncertainty of the Degradation Extractor. Furthermore, VDDC block integrates degradation information through dynamic convolution. Rhe VDDC also employs an Adaptive Intensity Scaling operation that adaptively adjusts the degradation representation according to the network hierarchy, thereby facilitating the effective integration of degradation information. Quantitative and qualitative experiments affirm the superiority of our approach. Copyright © 2024, The Authors. All rights reserved.

关键词： Convolution

来源：评论

学校读者我要写书评

暂无评论

A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis

arXiv

引用

arXiv 2024年

作者： Hei, Nailei Guo, Qianyu Wang, Zihao Wang, Yan Wang, Haofen Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University China Engineering Research Center of Robotics Ministry of Education Academy for Engineering & Technology Fudan University China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University China Tongji University China College of Design and Innovation Tongji University China

Well-designed prompts have demonstrated the potential to guide text-to-image models in generating amazing images. Although existing prompt engineering methods can provide high-level guidance, it is challenging for novice users to achieve the desired results by manually entering prompts due to a discrepancy between novice-user-input prompts and the model-preferred prompts. To bridge the distribution gap between user input behavior and model training datasets, we first construct a novel Coarse-Fine Granularity Prompts dataset (CFP) and propose a novel User-Friendly Fine-Grained Text Generation framework (UF-FGTG) for automated prompt optimization. For CFP, we construct a novel dataset for text-to-image tasks that combines coarse and fine-grained prompts to facilitate the development of automated prompt generation methods. For UF-FGTG, we propose a novel framework that automatically translates user-input prompts into model-preferred prompts. Specifically, we propose a prompt refiner that continually rewrites prompts to empower users to select results that align with their unique needs. Meanwhile, we integrate image-related loss functions from the text-to-image model into the training process of text generation to generate model-preferred prompts. Additionally, we propose an adaptive feature extraction module to ensure diversity in the generated results. Experiments demonstrate that our approach is capable of generating more visually appealing and diverse images than previous state-of-the-art methods, achieving an average improvement of 5% across six quality and aesthetic metrics. Data and code are available at https://***/Naylenv/UF-FGTG. Copyright © 2024, The Authors. All rights reserved.

关键词： Image enhancement

来源：评论

学校读者我要写书评

暂无评论

A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP

arXiv

引用

arXiv 2024年

作者： Tao, Zeng Wang, Yan Lin, Junxiong Wang, Haoran Mai, Xinji Yu, Jiawen Tong, Xuan Zhou, Ziheng Yan, Shaoqi Zhao, Qing Han, Liyuan Zhang, Wenqiang Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China School of Information Science and Technology Fudan University Shanghai China Institute of Automation Chinese Academy of Sciences Beijing China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

The performance of CLIP in dynamic facial expression recognition (DFER) task doesn’t yield exceptional results as observed in other CLIP-based classification tasks. While CLIP’s primary objective is to achieve alignment between images and text in the feature space, DFER poses challenges due to the abstract nature of text and the dynamic nature of video, making label representation limited and perfect alignment difficult. To address this issue, we have designed A3lign-DFER, which introduces a new DFER labeling paradigm to comprehensively achieve alignment, thus enhancing CLIP’s suitability for the DFER task. Specifically, our A3lign-DFER method is designed with multiple modules that work together to obtain the most suitable expanded-dimensional embeddings for classification and to achieve alignment in three key aspects: affective, dynamic, and bidirectional. We replace the input label text with a learnable Multi-Dimensional Alignment Token (MAT), enabling alignment of text to facial expression video samples in both affective and dynamic dimensions. After CLIP feature extraction, we introduce the Joint Dynamic Alignment Synchronizer (JAS), further facilitating synchronization and alignment in the temporal dimension. Additionally, we implement a Bidirectional Alignment Training Paradigm (BAP) to ensure gradual and steady training of parameters for both modalities. Our insightful and concise A3lign-DFER method achieves state-of-the-art results on multiple DFER datasets, including DFEW, FERV39k, and MAFW. Extensive ablation experiments and visualization studies demonstrate the effectiveness of A3lign-DFER. The code will be available in the future. © 2024, CC BY.

关键词： Alignment

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：