检索结果-内蒙古大学图书馆

UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity

学校读者我要写书评

暂无评论

UFineBench: Towards Text-based Person Retrieval with Ultra-f...

Conference on computer Vision and Pattern Recognition (CVPR)

作者： Jialong Zuo Hanyu Zhou Ying Nie Feng Zhang Tianyu Guo Nong Sang Yunhe Wang Changxin Gao National Key Laboratory of Multispectral Information Intelligent Processing Technology School of Artificial Intelligence and Automation Huazhong University of Science and Technology Huawei Noah's Ark Lab

ISBN: (数字)9798350353006

ISBN: (纸本)9798350353013

Existing text-based person retrieval datasets often have relatively coarse-grained text annotations. This hinders the model to comprehend the fine-grained semantics of query texts in real scenarios. To address this problem, we con-tribute a new benchmark named UFineBench for text-based person retrieval with ultra-fine granularity. Firstly, we construct a new dataset named UFine6926. We collect a large number of person images and manually annotate each image with two detailed textual descriptions, averaging 80.8 words each. The average word count is three to four times that of the previous datasets. In addition of standard in-domain evaluation, we also propose a spe-cial evaluation paradigm more representative of real sce-narios. It contains a new evaluation set with cross domains, cross textual granularity and cross textual styles, named UFine3C, and a new evaluation metric for accurately mea-suring retrieval ability, named mean Similarity Distribution (mSD). Moreover, we propose CFAM, a more efficient al-gorithm especially designed for text-based person retrieval with ultra fine-grained texts. It achieves fine granularity mining by adopting a shared cross-modal granularity de-coder and hard negative match mechanism. With standard in-domain evaluation, CFAM establishes competitive performance across various datasets, espe-cially on our ultra fine-grained UFine6926. Furthermore, by evaluating on UFine3C, we demonstrate that training on our UFine6926 significantly improves generalization to real scenarios compared with other coarse-grained datasets. The dataset and code will be made publicly available at https://***/Zplusdragon/UFineBench.

关键词： Training Measurement computer vision Correlation Codes Annotations Semantics

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Fu, Jiyuan Chen, Zhaoyu Jiang, Kaixun Guo, Haijing Wang, Jiafeng Gao, Shuyong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Despite the substantial advancements in Vision-Language Pre-training (VLP) models, their susceptibility to adversarial attacks poses a significant challenge. Existing work rarely studies the transferability of attacks on VLP models, resulting in a substantial performance gap from white-box attacks. We observe that prior work overlooks the interaction mechanisms between modalities, which plays a crucial role in understanding the intricacies of VLP models. In response, we propose a novel attack, called Collaborative Multimodal Interaction Attack (CMI-Attack), leveraging modality interaction through embedding guidance and interaction enhancement. Specifically, attacking text at the embedding level while preserving semantics, as well as utilizing interaction image gradients to enhance constraints on perturbations of texts and images. Significantly, in the image-text retrieval task on Flickr30K dataset, CMI-Attack raises the transfer success rates from ALBEF to TCL, CLIPViT and CLIPCNN by 8.11%-16.75% over state-of-the-art methods. Moreover, CMI-Attack also demonstrates superior performance in cross-task generalization scenarios. Our work addresses the underexplored realm of transfer attacks on VLP models, shedding light on the importance of modality interaction for enhanced adversarial robustness. Copyright © 2024, The Authors. All rights reserved.

关键词： Generative adversarial networks

C3-STISR: Scene Text Image Super-resolution with Triple Clues

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhao, Minyi Wang, Miao Bai, Fan Li, Bingjia Wang, Jie Zhou, Shuigeng Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai 200438 China ByteDance China

Scene text image super-resolution (STISR) has been regarded as an important pre-processing task for text recognition from low-resolution scene text images. Most recent approaches use the recognizer’s feedback as clue to guide super-resolution. However, directly using recognition clue has two problems: 1) Compatibility. It is in the form of probability distribution, has an obvious modal gap with STISR — a pixel-level task;2) Inaccuracy. it usually contains wrong information, thus will mislead the main task and degrade super-resolution performance. In this paper, we present a novel method C3-STISR that jointly exploits the recognizer’s feedback, visual and linguistical information as clues to guide super-resolution. Here, visual clue is from the images of texts predicted by the recognizer, which is informative and more compatible with the STISR task;while linguistical clue is generated by a pre-trained character-level language model, which is able to correct the predicted texts. We design effective extraction and fusion mechanisms for the triple cross-modal clues to generate a comprehensive and unified guidance for super-resolution. Extensive experiments on TextZoom show that C3-STISR outperforms the SOTA methods in fidelity and recognition performance. Code is available in https://***/zhaominyiz/C3-STISR. © 2022, CC BY-NC-SA.

关键词： Optical resolving power

Online Prototype Learning for Online Continual Learning

学校读者我要写书评

暂无评论

Online Prototype Learning for Online Continual Learning

International Conference on computer Vision (ICCV)

作者： Yujie Wei Jiaxin Ye Zhizhong Huang Junping Zhang Hongming Shan Institute of Science and Technology for Brain-Inspired Intelligence Fudan University Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University MOE Frontiers Center for Brain Science Fudan University Shanghai Center for Brain Science and Brain-Inspired Technology

Online continual learning (CL) studies the problem of learning continuously from a single-pass data stream while adapting to new data and mitigating catastrophic forgetting. Recently, by storing a small subset of old data, replay-based methods have shown promising performance. Unlike previous methods that focus on sample storage or knowledge distillation against catastrophic forgetting, this paper aims to understand why the online learning models fail to generalize well from a new perspective of shortcut learning. We identify shortcut learning as the key limiting factor for online CL, where the learned features may be biased, not generalizable to new tasks, and may have an adverse impact on knowledge distillation. To tackle this issue, we present the online prototype learning (OnPro) framework for online CL. First, we propose online prototype equilibrium to learn representative features against shortcut learning and discriminative features to avoid class confusion, ultimately achieving an equilibrium status that separates all seen classes well while learning new classes. Second, with the feedback of online prototypes, we devise a novel adaptive prototypical feedback mechanism to sense the classes that are easily misclassified and then enhance their boundaries. Extensive experimental results on widely-used benchmark datasets demonstrate the superior performance of OnPro over the state-of-the-art baseline methods. Source code is available at https://***/weilllllls/OnPro.

关键词：

Memory-based Message Passing: Decoupling the Message for Propogation from Discrimination

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Chen, Jie Liu, Weiqi Pu, Jian Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Fudan University Shanghai China

Message passing is a fundamental procedure for graph neural networks in the field of graph representation learning. Based on the homophily assumption, the current message passing always aggregates features of connected nodes, such as the graph Laplacian smoothing process. However, real-world graphs tend to be noisy and/or non-smooth. The homophily assumption does not always hold, leading to sub-optimal results. A revised message passing method needs to maintain each node's discriminative ability when aggregating the message from neighbors. To this end, we propose a Memory-based Message Passing (MMP) method to decouple the message of each node into a self-embedding part for discrimination and a memory part for propagation. Furthermore, we develop a control mechanism and a decoupling regularization to control the ratio of absorbing and excluding the message in the memory for each node. More importantly, our MMP is a general skill that can work as an additional layer to help improve traditional GNNs performance. Extensive experiments on various datasets with different homophily ratios demonstrate the effectiveness and robustness of the proposed method. © 2022, CC BY.

关键词： Message passing

General Compression Framework for Efficient Transformer Object Tracking

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hong, Lingyi Li, Jinglun Zhou, Xinyu Yan, Shilin Guo, Pinxue Jiang, Kaixun Chen, Zhaoyu Gao, Shuyong Zhang, Wei Lu, Hong Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China

Transformer-based trackers have established a dominant role in the field of visual object tracking. While these trackers exhibit promising performance, their deployment on resource-constrained devices remains challenging due to inefficiencies. To improve the inference efficiency and reduce the computation cost, prior approaches have aimed to either design lightweight trackers or distill knowledge from larger teacher models into more compact student trackers. However, these solutions often sacrifice accuracy for speed. Thus, we propose a general model compression framework for efficient transformer object tracking, named CompressTracker, to reduce the size of a pre-trained tracking model into a lightweight tracker with minimal performance degradation. Our approach features a novel stage division strategy that segments the transformer layers of the teacher model into distinct stages, enabling the student model to emulate each corresponding teacher stage more effectively. Additionally, we also design a unique replacement training technique that involves randomly substituting specific stages in the student model with those from the teacher model, as opposed to training the student model in isolation. Replacement training enhances the student model’s ability to replicate the teacher model’s behavior. To further forcing student model to emulate teacher model, we incorporate prediction guidance and stage-wise feature mimicking to provide additional supervision during the teacher model’s compression process. Our framework CompressTracker is structurally agnostic, making it compatible with any transformer architecture. We conduct a series of experiment to verify the effectiveness and generalizability of CompressTracker. Our CompressTracker-4 with 4 transformer layers, which is compressed from OSTrack, retains about 96% performance on LaSOT (66.1% AUC) while achieves 2.17× speed up. © 2024, CC BY.

关键词： Students

P3S-Diffusion:A Selective Subject-driven Generation Framework via Point Supervision

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Hu, Junjie Gao, Shuyong Hong, Lingyi Wang, Qishan Zhao, Yuzhou Wang, Yan Zhang, Wenqiang Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China Shanghai Engineering Research Center of AI & Robotics Academy for Engineering & Technology Fudan University Shanghai China Engineering Research Center of AI & Robotics Ministry of Education Academy for Engineering & Technology Fudan University Shanghai China

Recent research in subject-driven generation increasingly emphasizes the importance of selective subject features. Nevertheless, accurately selecting the content in a given reference image still poses challenges, especially when selecting the similar subjects in an image (e.g., two different dogs). Some methods attempt to use text prompts or pixel masks to isolate specific elements. However, text prompts often fall short in precisely describing specific content, and pixel masks are often expensive. To address this, we introduce P3S-Diffusion, a novel architecture designed for context-selected subject-driven generation via point supervision. P3S-Diffusion leverages minimal cost label (e.g., points) to generate subject-driven images. During fine-tuning, it can generate an expanded base mask from these points, obviating the need for additional segmentation models. The mask is employed for inpainting and aligning with subject representation. The P3S-Diffusion preserves fine features of the subjects through Multi-layers Condition Injection. Enhanced by the Attention Consistency Loss for improved training, extensive experiments demonstrate its excellent feature preservation and image generation capabilities. Copyright © 2024, The Authors. All rights reserved.

关键词： Pixels

Deep Representation Learning for Electron Ionization Mass Spectra Retrieval

学校读者我要写书评

暂无评论

Deep Representation Learning for Electron Ionization Mass Sp...

IEEE International Conference on Bioinformatics and Biomedicine (BIBM)

作者： Shibo Liu Xuan Zhang Anlei Jiao Shiwei Sun Longyang Dian Xuefeng Cui School of Computer Science and Technology Shandong University Qingdao China Key Lab of Intelligent Information Processing Institute of Computing Technology Chinese Academy of Sciences Beijing China State Key Laboratory of Microbial Technology Shandong University Qingdao China

ISBN: (数字)9798350386226

ISBN: (纸本)9798350386233

The task of retrieving and analyzing mass spectra is indispensable for the identification of compounds in mass spectrometry (MS). This methodology is of critical importance as it enables researchers to correlate observed spectra with established databases, thereby precisely determining the chemical composition of samples. The primary challenges to its efficacy lie in optimizing the balance between retrieval accuracy and processing speed. Empirical studies have demonstrated that by converting mass spectra into embeddings via deep learning, it is possible to achieve both high accuracy and speed in retrieval. Nevertheless, there are complex challenges associated with employing deep learning for spectral embedding, particularly within the domain of electron ionization mass spectrometry (EI-MS). In this paper, we introduce a novel representation learning technique termed EI-MS2VEC for EI-MS retrieval. Our spectrum retrieval methodology surpasses current state-of-the-art techniques such as FastEI. For the in-silico library, we attain hit rate@1 and hit rate@10 of 43.6% and 84.5%, respectively, compared to FastEI’s 36.7% and 80.4%. Moreover, our retrieval approach operates with an order of magnitude greater speed than FastEI. The source code is available on Github (https://***/xfcui/EI-MS2VEC).

关键词： Representation learning Deep learning Accuracy Databases Source coding Ionization Predictive models Mass spectroscopy Electrons Software development management

From Efficient Multimodal Models to World Models: A Survey

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Mai, Xinji Tao, Zeng Lin, Junxiong Wang, Haoran Chang, Yang Kang, Yanlan Wang, Yan Zhang, Wenqiang Shanghai Engineering Research Center of AI and Robotics Academy for Engineering and Technology Fudan University Shanghai China Engineering Research Center of AI and Robotics Ministry of Education Academy for Engineering and Technology Fudan University Shanghai China Shanghai Key Lab of Intelligent Information Processing School of Computer Science Fudan University Shanghai China

Multimodal Large Models (MLMs) are becoming a significant research focus, combining powerful large language models with multimodal learning to perform complex tasks across different data modalities. This review explores the latest developments and challenges in MLMs, emphasizing their potential in achieving artificial general intelligence and as a pathway to world models. We provide an overview of key techniques such as Multimodal Chain of Thought (M-COT), Multimodal Instruction Tuning (M-IT), and Multimodal In-Context Learning (M-ICL). Additionally, we discuss both the fundamental and specific technologies of multimodal models, highlighting their applications, input/output modalities, and design characteristics. Despite significant advancements, the development of a unified multimodal model remains elusive. We discuss the integration of 3D generation and embodied intelligence to enhance world simulation capabilities and propose incorporating external rule systems for improved reasoning and decision-making. Finally, we outline future research directions to address these challenges and advance the field. Copyright © 2024, The Authors. All rights reserved.

关键词： Decision making