检索结果-内蒙古大学图书馆

Attention head purification: A new perspective to harness CLIP for domain generalization

IMAGE AND vision COMPUTING 2025年 157卷

作者： Wang, Yingfan Kang, Guoliang Beihang Univ 37 Xueyuan Rd Beijing 100191 Peoples R China

Domain Generalization (DG) aims to learn a model from multiple source domains to achieve satisfactory performance on unseen target domains. Recent works introduce CLIP to DG tasks due to its superior image-text alignment and zeros-shot performance. Previous methods either utilize full fine-tuning or prompt-learning paradigms to harness CLIP for DG tasks. Those works focus on avoiding catastrophic forgetting of the original knowledge encoded in CLIP but ignore that the knowledge encoded in CLIP in nature may contain domain-specific cues that constrain its domain generalization performance. In this paper, we propose a new perspective to harness CLIP for DG, i.e., attention head purification. We observe that different attention heads may encode different properties of an image and selecting heads appropriately may yield remarkable performance improvement across domains. Based on such observations, we purify the attention heads of CLIP from two levels, including task-level purification and domain-level purification. For task-level purification, we design head-aware LoRA to make each head more adapted to the task we considered. For domain-level purification, we perform head selection via a simple gating strategy. We utilize MMD loss to encourage masked head features to be more domain-invariant to emphasize more generalizable properties/heads. During training, we jointly perform task-level purification and domain-level purification. We conduct experiments on various representative DG benchmarks. Though simple, extensive experiments demonstrate that our method performs favorably against previous state-of-the-arts.

关键词： Multi-source domain generalization vision-language model Low-rank adaptation

来源：评论

学校读者我要写书评

暂无评论

Inference Calibration of vision-language Foundation models for Zero-Shot and Few-Shot Learning

引用

PATTERN RECOGNITION LETTERS 2025年 192卷 15-21页

作者： Hu, Minyang Chang, Hong Shan, Shiguang Chen, Xilin Chinese Acad Sci Chinese Acad Sci CAS Inst Comp Technol Key Lab Intelligent Informat Proc Beijing 100190 Peoples R China Univ Chinese Acad Sci Beijing 100049 Peoples R China

Contrastive language-Image Pre-training (CLIP) models exhibit impressive zero-shot performance across various downstream cross-modal tasks by simply computing the dot product between image and text features. CLIP is pre-trained on large-scale image-text pairs using the InfoNCE loss, which maximizes the cosine similarity of positive image-text pairs while minimizing the similarity of negative pairs. However, an objective mismatch exists between the downstream usage and the pre-training phase, as the inference phase fails to exploit information from negative samples. Intuitively, since the CLIP model has been optimized based on the InfoNCE loss, the downstream usage should also be in alignment. In this paper, we start from analyzing the InfoNCE loss and derive its upper bound. Our derivation reveals that the dot-product operation serves a zero-order approximation of this upper bound, while a centralization operation represents a first-order approximation. To address the objective mismatch problem, we propose a novel method, Inference Calibration (IC), which leverages the first-order and second-order moments of data distribution to calibrate features for zero-shot and few-shot scenarios. Experiments on various cross-modal tasks demonstrate the effectiveness of IC in both zero-shot and few-shot scenarios over dot-product operation and other comparative methods.

关键词： vision-language model Zero/Few-Shot Learning Contrastive language-Image Pre-training

来源：评论

学校读者我要写书评

暂无评论

ItpCtrl-AI: End-to-end interpretable and controllable artificial intelligence by modeling radiologists' intentions☆

引用

ARTIFICIAL INTELLIGENCE IN MEDICINE 2025年 160卷 103054-103054页

作者： Pham, Trong-Thang Brecheisen, Jacob Wu, Carol C. Nguyen, Hien Deng, Zhigang Adjeroh, Donald Doretto, Gianfranco Choudhary, Arabinda Le, Ngan Univ Arkansas Dept EECS AICV Lab Little Rock AR 72701 USA MD Anderson Canc Ctr Houston TX 77079 USA Univ Houston Dept ECE Houston TX 77204 USA Univ Houston Dept CS Houston TX 77204 USA West Virginia Univ Dept CSEE Morgantown WV 26506 USA Univ Arkansas Med Sci Little Rock AR 72705 USA

Using Deep Learning in computer-aided diagnosis systems has been of great interest due to its impressive performance in the general domain and medical domain. However, a notable challenge is the lack of explainability of many advanced models, which poses risks in critical applications such as diagnosing findings in CXR. To address this problem, we propose ItpCtrl-AI, a novel end-to-end interpretable and controllable framework that mirrors the decision-making process of the radiologist. By emulating the eye gaze patterns of radiologists, our framework initially determines the focal areas and assesses the significance of each pixel within those regions. Asa result, the model generates an attention heatmap representing radiologists' attention, which is then used to extract attended visual information to diagnose the findings. By allowing the directional input, our framework is controllable by the user. Furthermore, by displaying the eye gaze heatmap which guides the diagnostic conclusion, the underlying rationale behind the model's decision is revealed, thereby making it interpretable. In addition to developing an interpretable and controllable framework, our work includes the creation of a dataset, named Diagnosed-Gaze++, which aligns medical findings with eye gaze data. Our extensive experimentation validates the effectiveness of our approach in generating accurate attention heatmaps and diagnoses. The experimental results show that our model not only accurately identifies medical findings but also precisely produces the eye gaze attention of radiologists. The dataset, models, and source code will be made publicly available upon acceptance.

关键词： Interpretable deep learning Computer-aided diagnosis vision-language model Radiology Radiologist's intention Gaze intention

来源：评论

学校读者我要写书评

暂无评论

When Adversarial Training Meets Prompt Tuning: Adversarial Dual Prompt Tuning for Unsupervised Domain Adaptation

引用

IEEE TRANSACTIONS ON IMAGE PROCESSING 2025年 34卷 1427-1440页

作者： Cui, Chaoran Liu, Ziyi Gong, Shuai Zhu, Lei Zhang, Chunyun Liu, Hui Shandong Univ Finance & Econ Sch Comp & Artificial Intelligence Jinan 250014 Peoples R China Tongji Univ Coll Elect & Informat Engn Shanghai 201804 Peoples R China

Unsupervised domain adaptation (UDA) aims to adapt models learned from a well-annotated source domain to a target domain, where only unlabeled samples are available. To this end, adversarial training is widely used in conventional UDA methods to reduce the discrepancy between source and target domains. Recently, prompt tuning has emerged as an efficient way to adapt large pre-trained vision-language models like CLIP to a variety of downstream tasks. In this paper, we present a novel method named Adversarial DuAl Prompt Tuning (ADAPT) for UDA, which employs text prompts and visual prompts to guide CLIP simultaneously. Rather than simply performing a joint optimization of text prompts and visual prompts, we integrate text prompt tuning and visual prompt tuning into a collaborative framework where they engage in an adversarial game: text prompt tuning focuses on distinguishing between source and target images, whereas visual prompt tuning seeks to align source and target domains. Unlike most existing adversarial training-based UDA approaches, ADAPT does not require explicit domain discriminators for domain alignment. Instead, the objective is effectively achieved at both global and category levels through modeling the joint probability distribution of images on domains and categories. Extensive experiments on four benchmark datasets demonstrate the effectiveness of our ADAPT method for UDA.

关键词： Tuning Visualization Adaptation models Training Feature extraction Transformers Probability distribution Text recognition Target recognition Image recognition Unsupervised domain adaptation adversarial training prompt tuning vision-language model

来源：评论

学校读者我要写书评

暂无评论

Transferable Unintentional Action Localization With language-Guided Intention Translation

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2025年第5期47卷 3863-3877页

作者： Xu, Jinglin Rao, Yongming Zhou, Jie Lu, Jiwen Univ Sci & Technol Sch Intelligence Sci & Technol Beijing 100083 Peoples R China Tsinghua Univ Dept Automat Beijing 100084 Peoples R China

Unintentional action localization (UAL) is a challenging task that requires reasoning about action intention clues to detect the temporal locations of unintentional action occurrences in real-world videos. Previous efforts usually treated this task as a dense binary classification problem and did not fully explore the relationships between intention clues and unintentional actions, resulting in unsatisfactory performance on open-set scenarios during inference. In this paper, we propose a Transferable Unintentional Action Localization framework by introducing language-guided intention translation, which explicitly formulates unintentional action localization as an open-set localization problem. Our framework constructs a transferable reasoning model guided by natural languages to translate the action intention of the entire video, which generates natural and powerful supervision signals for reconstructing complete action intention clues to address the problem of unintentional action localization. Based on the fact that a video with failure action is composed of intentional and unintentional parts connected by a transient action transition. Our transferable reasoning model employs a transformer architecture to transfer knowledge between intentional and unintentional parts for learning complementary semantic representations of these two parts, completing the action intention clue in an implicit supervision manner. We also present a dense voting scheme for detecting the action transition from intentional to unintentional using discriminative representations incorporating action intention clues. Extensive experiments demonstrate that our framework outperforms representative unintentional action localization methods in a wide range of open-set scenarios. In addition, we create a new unintentional sports video dataset, FS-Falls, and extend our framework from in-the-wild scenarios to competitive sports to demonstrate better generalization ability. We hope this work will pro

关键词： Videos Location awareness Cognition Visualization Sports Semantics Translation Training Feature extraction Computational modeling Unintentional action localization action intention vision-language model intention translation generalization ability

来源：评论

学校读者我要写书评

暂无评论

Class Concept Representation From Contextual Texts for Training-Free Multi-Label Recognition

引用

IEEE ACCESS 2025年 13卷 36965-36977页

作者： Kang, Dong Un Lee, Hyunwoo Chun, Se Young Seoul Natl Univ Dept Elect & Comp Engn Seoul 08826 South Korea Seoul Natl Univ INMC Seoul 08826 South Korea Seoul Natl Univ IPAI Seoul 08826 South Korea

The power of large vision-language models (VLMs) has been demonstrated for downstream vision tasks, including multi-label recognition (MLR) with a training-free approach or prompt tuning by measuring the cosine similarity between the text features related to class names and the visual features of the images. Previous studies typically represent class-related text features by averaging simple handcrafted prompts with class names (e.g., "a photo of class name"), which lack the contextual information required to effectively handle diverse content and capture the co-occurrence of multiple semantic classes within an image. In addition, prompt tuning inherently requires labeled data or additional training, making it susceptible to overfitting for context tokens and hindering generalization. To address these limitations, we propose a training-free and label-free MLR framework that leverages the abundant text descriptions of novel classes to narrow the gap between the visual and text features at inference. Inspired by how humans form the concepts on words using contexts or the patterns of co-occurrence with other words, we propose a class concept representation for zero-shot MLR for large VLMs, which the rich contextual information embedded in large-scale image descriptions (e.g., "A person holding a large pair of scissors"). To further improve the alignment of the visual features of VLMs to the class concept representation, we also present a context-guided visual representation based on an attention process that operates in the same linear space as the class concept representation. Experimental results obtained on diverse benchmarks demonstrate that the proposed methods substantially improve the performance of existing zero-shot MLR methods, achieving an average improvement of 9.4% in mean average precision (mAP) over the zero-shot contrastive language-image pretraining (CLIP) method and 2.2% in mAP over the TaI-DPT zero-shot prompt-tuning method. In addition, our proposed

关键词： Tuning Text recognition Image recognition Visualization Training Costs Vectors Annotations Volume measurement Solid modeling Zero-shot multi-label recognition vision-language model class concept contextual text

来源：评论

学校读者我要写书评

暂无评论

GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detection

引用

INTERNATIONAL JOURNAL OF COMPUTER vision 2025年第6期133卷 3586-3596页

作者： Miyai, Atsuyuki Yu, Qing Irie, Go Aizawa, Kiyoharu Univ Tokyo Tokyo Japan LY Corp Tokyo Japan Tokyo Univ Sci Tokyo Japan

Zero-shot OOD detection is a task that detects OOD images during inference with only in-distribution (ID) class names. Existing methods assume ID images contain a single, centered object, and do not consider the more realistic multi-object scenarios, where both ID and OOD objects are present. To meet the needs of many users, the detection method must have the flexibility to adapt the type of ID images. To this end, we present Global-Local Maximum Concept Matching (GL-MCM), which incorporates local image scores as an auxiliary score to enhance the separability of global and local visual features. Due to the simple ensemble score function design, GL-MCM can control the type of ID images with a single weight parameter. Experiments on ImageNet and multi-object benchmarks demonstrate that GL-MCM outperforms baseline zero-shot methods and is comparable to fully supervised methods. Furthermore, GL-MCM offers strong flexibility in adjusting the target type of ID images. The code is available via https://***/AtsuMiyai/GL-MCM.

关键词： CLIP vision-language model Zero-shot prediction Multi-modality Out-of-distribution detection

来源：评论

学校读者我要写书评

暂无评论

Learning Semantic-aware Representation in Visual-language models for Multi-label Recognition with Partial Labels

引用

ACM TRANSACTIONS ON MULTIMEDIA COMPUTING COMMUNICATIONS AND APPLICATIONS 2025年第3期21卷 1-19页

作者： Ruan, Haoxian Xu, Zhihua Yang, Zhijing Lu, Yongyi Qin, Jinghui Chen, Tianshui Guangdong Univ Technol Guangzhou Peoples R China Guangdong Univ Technol Informat Engn Guangzhou Peoples R China

Multi-label recognition with partial labels (MLR-PL), in which only some labels are known while others are unknown for each image, is a practical task in computer vision, since collecting large-scale and complete multi-label datasets is difficult in real application scenarios. Recently, vision language models (e.g., CLIP) have demonstrated impressive transferability to downstream tasks in data limited or label limited settings. However, current CLIP-based methods suffer from semantic confusion in MLR task due to the lack of fine-grained information in the single global visual and textual representation for all categories. In this work, we address this problem by introducing a semantic decoupling module and a category-specific prompt optimization method in CLIP-based framework. Specifically, the semantic decoupling module following the visual encoder learns category-specific feature maps by utilizing the semantic-guided spatial attention mechanism. Moreover, the category-specific prompt optimization method is introduced to learn text representations aligned with category semantics. Therefore, the prediction of each category is independent, which alleviate the semantic confusion problem. Extensive experiments on Microsoft COCO 2014 and Pascal VOC 2007 datasets demonstrate that the proposed framework significantly outperforms current state-of-art methods with a simpler model structure. Additionally, visual analysis shows that our method effectively separates information from different categories and achieves better performance compared to CLIP-based baseline method.

关键词： Multi-label recognition Partial labels vision-language model

来源：评论

学校读者我要写书评

暂无评论

Bi-VLGM: Bi-Level Class-Severity-Aware vision-language Graph Matching for Text Guided Medical Image Segmentation

引用

INTERNATIONAL JOURNAL OF COMPUTER vision 2025年第3期133卷 1375-1391页

作者： Chen, Wenting Liu, Jie Liu, Tianming Yuan, Yixuan City Univ Hong Kong Dept Elect Engn Kowloon Hong Kong Peoples R China Chinese Univ Hong Kong Dept Elect Engn Sha Tin Hong Kong Peoples R China Univ Georgia Dept Comp Sci Athens GA USA

Medical reports containing specific diagnostic results and additional information not present in medical images can be effectively employed to assist image understanding tasks, and the modality gap between vision and language can be bridged by vision-language matching (VLM). However, current vision-language models distort the intra-model relation and only include class information in reports that is insufficient for segmentation task. In this paper, we introduce a novel Bi-level class-severity-aware vision-language Graph Matching (Bi-VLGM) for text guided medical image segmentation, composed of a word-level VLGM module and a sentence-level VLGM module, to exploit the class-severity-aware relation among visual-textual features. In word-level VLGM, to mitigate the distorted intra-modal relation during VLM, we reformulate VLM as graph matching problem and introduce a vision-language graph matching (VLGM) to exploit the high-order relation among visual-textual features. Then, we perform VLGM between the local features for each class region and class-aware prompts to bridge their gap. In sentence-level VLGM, to provide disease severity information for segmentation task, we introduce a severity-aware prompting to quantify the severity level of disease lesion, and perform VLGM between the global features and the severity-aware prompts. By exploiting the relation between the local (global) and class (severity) features, the segmentation model can include the class-aware and severity-aware information to promote segmentation performance. Extensive experiments proved the effectiveness of our method and its superiority to existing methods. The source code will be released.

关键词： Medical image segmentation Text guided segmentation Graph matching vision-language model

来源：评论

学校读者我要写书评

暂无评论

Enhancing cross-domain generalization by fusing language-guided feature remapping

引用

INFORMATION FUSION 2025年 119卷

作者： Qiao, Ziteng Shi, Dianxi Jin, Songchang Shi, Yanyan Jing, Luoxi Qiu, Chunping Acad Mil Sci Beijing 100071 Peoples R China Natl Univ Def Technol Coll Comp Changsha 410073 Peoples R China

Domain generalization refers to training a model with annotated source domain data and making it generalize to various unseen target domains. It has been extensively studied in classification, but remains challenging in object detection. Existing domain generalization object detection methods mainly rely on generative or adversarial data augmentation, which increases the complexity of training. Recently, vision-language models (VLMs), such as CLIP, have demonstrated strong cross-modal alignment capabilities, showing potential for enhancing domain generalization. On this basis, the paper proposes a language-guided feature remapping method, which leverages VLMs to augment sample features and improve the generalization performance of regular models. In detail, we first construct a teacher-student network structure. Then, we introduce a feature remapping module that remaps sample features in both local and global spatial dimensions to improve the distribution of feature representations. Concurrently, we design domain prompts and class prompts to guide the sample features to remap into amore generalized and universal feature space. Finally, we establish a knowledge distillation structure to facilitate knowledge transfer between teacher and student networks, enhancing the domain generalization ability of the student network. Multiple experimental results demonstrate the superiority of our proposed method.

关键词： Domain generalization Multimodal fusion vision-language model Object detection

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：