检索结果-内蒙古大学图书馆

Model LEGO: creating models like disassembling and assembling building blocks 24

学校读者我要写书评

暂无评论

Model LEGO: creating models like disassembling and assemblin...

Proceedings of the 38th International Conference on Neural Information Processing Systems

作者： Jiacong Hu Jing Gao Jingwen Ye Yang Gao Xingen Wang Zunlei Feng Mingli Song College of Computer Science and Technology Zhejiang University and State Key Laboratory of Blockchain and Data Security Zhejiang University Robotics Institute School of Computer Science Carnegie Mellon University Electrical and Computer Engineering National University of Singapore Bangsheng Technology Co. Ltd. College of Computer Science and Technology Zhejiang University and Bangsheng Technology Co. Ltd. School of Software Technology Zhejiang University and State Key Laboratory of Blockchain and Data Security Zhejiang University and Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain College of Computer Science and Technology Zhejiang University and State Key Laboratory of Blockchain and Data Security Zhejiang University and Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security

ISBN: (纸本)9798331314385

With the rapid development of deep learning, the increasing complexity and scale of parameters make training a new model increasingly resource-intensive. In this paper, we start from the classic convolutional neural network (CNN) and explore a paradigm that does not require training to obtain new models. Similar to the birth of CNN inspired by receptive fields in the biological visual system, we draw inspiration from the information subsystem pathways in the biological visual system and propose Model Disassembling and Assembling (MDA). During model disassembling, we introduce the concept of relative contribution and propose a component locating technique to extract task-aware components from trained CNN classifiers. For model assembling, we present the alignment padding strategy and parameter scaling strategy to construct a new model tailored for a specific task, utilizing the disassembled task-aware components. The entire process is akin to playing with LEGO bricks, enabling arbitrary assembly of new models, and providing a novel perspective for model creation and reuse. Extensive experiments showcase that task-aware components disassembled from CNN classifiers or new models assembled using these components closely match or even surpass the performance of the baseline, demonstrating its promising results for model reuse. Furthermore, MDA exhibits diverse potential applications, with comprehensive experiments exploring model decision route analysis, model compression, knowledge distillation, and more. For more information, please visit https://***/.

关键词：

Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Xu Ma, Jiaqi Wang, Guoli Zhang, Qian Zhang, Huan Zhang, Lefei The Institute of Artificial Intelligence School of Computer Science Wuhan University Wuhan430072 China The Hubei Luojia Laboratory Wuhan China The Horizon Robotics Beijing100083 China The School of Information Engineering Guangdong University of Technology Guangzhou510006 China

The limitations of task-specific and general image restoration methods for specific degradation have prompted the development of all-in-one image restoration techniques. However, the diversity of patterns among multiple degradation, along with the significant uncertainties in mapping between degraded images of different severities and their corresponding undistorted versions, pose significant challenges to the all-in-one restoration tasks. To address these challenges, we propose Perceive-IR, an all-in-one image restorer designed to achieve fine-grained quality control that enables restored images to more closely resemble their undistorted counterparts, regardless of the type or severity of degradation. Specifically, Perceive-IR contains two stages: (1) prompt learning stage and (2) restoration stage. In the prompt learning stage, we leverage prompt learning to acquire a fine-grained quality perceiver capable of distinguishing three-tier quality levels by constraining the prompt-image similarity in the CLIP perception space. Subsequently, this quality perceiver and difficulty-adaptive perceptual loss are integrated as a quality-aware learning strategy to realize fine-grained quality control in restoration stage. For the restoration stage, a semantic guidance module (SGM) and compact feature extraction (CFE) are proposed to further promote the restoration process by utilizing the robust semantic information from the pre-trained large scale vision models and distinguishing degradation-specific features. Extensive experiments demonstrate that our Perceive-IR outperforms state-of-the-art methods in all-in-one image restoration tasks and exhibit superior generalization ability when dealing with unseen tasks. © 2024, CC BY.

关键词： Image reconstruction

P3S-Diffusion:A Selective Subject-driven Generation Framework via Point Supervision

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hu, Junjie Gao, Shuyong Hong, Lingyi Wang, Qishan Zhao, Yuzhou Wang, Yan Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Recent research in subject-driven generation increasingly emphasizes the importance of selective subject features. Nevertheless, accurately selecting the content in a given reference image still poses challenges, especially when selecting the similar subjects in an image (e.g., two different dogs). Some methods attempt to use text prompts or pixel masks to isolate specific elements. However, text prompts often fall short in precisely describing specific content, and pixel masks are often expensive. To address this, we introduce P3S-Diffusion, a novel architecture designed for context-selected subject-driven generation via point supervision. P3S-Diffusion leverages minimal cost label (e.g., points) to generate subject-driven images. During fine-tuning, it can generate an expanded base mask from these points, obviating the need for additional segmentation models. The mask is employed for inpainting and aligning with subject representation. The P3S-Diffusion preserves fine features of the subjects through Multi-layers Condition Injection. Enhanced by the Attention Consistency Loss for improved training, extensive experiments demonstrate its excellent feature preservation and image generation capabilities. Copyright © 2024, The Authors. All rights reserved.

关键词： Pixels

General Compression Framework for Efficient Transformer Object Tracking

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hong, Lingyi Li, Jinglun Zhou, Xinyu Yan, Shilin Guo, Pinxue Jiang, Kaixun Chen, Zhaoyu Gao, Shuyong Zhang, Wei Lu, Hong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China

Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model’s ability to replicate the teacher model’s behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model’s compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17× speed up. © 2024, CC BY.

关键词： Students

Human-machine augmented intelligence:research and applications

学校读者我要写书评

暂无评论

Frontiers of Information technology & Electronic Engineering 2022年第8期23卷 1139-1141页

作者： Jianru XUE Bin HU Lingxi LI Junping ZHANG Institute of Artificial Intelligence and Robotics Xi’an Jiaotong UniversityXi'an 710049China School of Information Science and Engineering Lanzhou UniversityLanzhou 730099China Department of Electrical and Computer Engineering Purdue School of Engineering and TechnologyIndiana University-Purdue University IndianapolisIN 46202-5195USA School of Computer Science Fudan UniversityShanghai 200433China

Current research on artificial intlligence(Al)has been entering a new era,with AI technologies and Al-enabled applications emerging in almost every aspect of human ***,avoiding the risk caused by limitations of AI technologies has becomea grand *** main idea of human-machine augmented intelligence(HAlI)is to adopt the role of humans or to embed human-like cognitive abilities into intelligent *** atention and efforts from academia,industry,and governments are attracted by the HAl idea,whose effects are far-reaching.

关键词： intelligence Human limitations

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Fu, Jiyuan Chen, Zhaoyu Jiang, Kaixun Guo, Haijing Wang, Jiafeng Gao, Shuyong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging modality interaction through embedding guidance and interaction enhancement. Specifically, attacking text at the embedding level while preserving semantics, as well as utilizing interaction image gradients to enhance constraints on perturbations of texts and images. Significantly, in the image-text retrieval task on Flickr30K dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL, CLIPViT and CLIPCNN by 8.11%-16.75% over state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior performance in cross-task generalization scenarios. Our work addresses the underexplored realm of transfer attacks on VLP models, shedding light on the importance of modality interaction for enhanced adversarial robustness. Copyright © 2024, The Authors. All rights reserved.

关键词： Generative adversarial networks

EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Lin, Jiacheng Chen, Jiajun Peng, Kunyu He, Xuan Li, Zhiyong Stiefelhagen, Rainer Yang, Kailun The College of Computer Science and Electronic Engineering Hunan University Changsha410082 China The School of Robotics Hunan University Changsha410012 China The National Engineering Research Center of Robot Visual Perception and Control Technology Hunan University Changsha410082 China The Institute for Robotics and An-Thropomatics Karlsruhe Institute of Technology Karlsruhe76131 Germany

This paper introduces the task of Auditory Referring Multi-Object Tracking (AR-MOT), which dynamically tracks specific objects in a video sequence based on audio expressions and appears as a challenging problem in autonomous driving. Due to the lack of semantic modeling capacity in audio and video, existing works have mainly focused on text-based multiobject tracking, which often comes at the cost of tracking quality, interaction efficiency, and even the safety of assistance systems, limiting the application of such methods in autonomous driving. In this paper, we delve into the problem of AR-MOT from the perspective of audio-video fusion and audio-video tracking. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. The dual streams are intertwined with our Bidirectional Frequency-domain Cross-attention Fusion Module (Bi-FCFM), which bidirectionally fuses audio and video features from both frequency- and spatiotemporal domains. Moreover, we propose the Audio-visual Contrastive Tracking Learning (ACTL) regime to extract homogeneous semantic features between expressions and visual objects by learning homogeneous features between different audio and video objects effectively. Aside from the architectural design, we establish the first set of large-scale AR-MOT benchmarks, including Echo-KITTI, Echo-KITTI+, and Echo-BDD. Extensive experiments on the established benchmarks demonstrate the effectiveness of the proposed EchoTrack and its components. The source code and datasets are available at https://***/lab206/EchoTrack. Copyright © 2024, The Authors. All rights reserved.

关键词： Semantics

From Efficient Multimodal Models to World Models: A Survey

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Mai, Xinji Tao, Zeng Lin, Junxiong Wang, Haoran Chang, Yang Kang, Yanlan Wang, Yan Zhang, Wenqiang Shanghai Engineering Research Center of AI and Robotics Academy for Engineering and Technology Fudan University Shanghai China Engineering Research Center of AI and Robotics Ministry of Education Academy for Engineering and Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field. Copyright © 2024, The Authors. All rights reserved.

关键词： Decision making

3D Facial Expressions through Analysis-by-Neural-Synthesis

学校读者我要写书评

暂无评论

3D Facial Expressions through Analysis-by-Neural-Synthesis

Conference on computer Vision and Pattern Recognition (CVPR)

作者： George Retsinas Panagiotis P. Filntisis Radek Daněček Victoria F. Abrevaya Anastasios Roussos Timo Bolkarr Petros Maragos Institute of Robotics Athena Research Center Maroussi Greece MPI for Intelligent Systems Tübingen Germany Institute of Computer Science (ICS) Foundation for Research & Technology - Hellas (FORTH) Greece School of Electrical & Computer Engineering National Technical University of Athens Greece

ISBN: (数字)9798350353006

ISBN: (纸本)9798350353013

While existing methods for 3D face reconstruction from in-the-wild images excel at recovering the overall face shape, they commonly miss subtle, extreme, asymmetric, or rarely observed expressions. We improve upon these meth-ods with SMIRK (Spatial Modeling for Image-based Reconstruction of Kinesics), which faithfully reconstructs expres-sive 3D faces from images. We identify two key limitations in existing methods: shortcomings in their self-supervised training formulation, and a lack of expression diversity in the training images. For training, most methods employ differentiable rendering to compare a predicted face mesh with the input image, along with a plethora of additional loss functions. This differentiable rendering loss not only has to provide supervision to optimize for 3D face geom-etry, camera, albedo, and lighting, which is an ill-posed optimization problem, but the domain gap between ren-dering and input image further hinders the learning pro-cess. Instead, SMIRK replaces the differentiable rendering with a neural rendering module that, given the ren-dered predicted mesh geometry, and sparsely sampled pix-els of the input image, generates a face image. As the neural rendering gets color information from sampled im-age pixels, supervising with neural rendering-based reconstruction loss can focus solely on the geometry. Further it enables us to generate images of the input identity with varying expressions while training. These are then utilized as input to the reconstruction model and used as supervision with ground truth geometry. This effectively augments the training data and enhances the generalization for di-verse expressions. Our qualitative, quantitative and partic-ularly our perceptual evaluations demonstrate that SMIRK achieves the new state-of-the art performance on accurate expression reconstruction. For our method's source code, demo video and more, please visit our project webpage: https://***/smirk/.

关键词： Training geometry Solid modeling Three-dimensional displays Accuracy Face recognition Training data