检索结果-内蒙古大学图书馆

language-Driven Visual Consensus for Zero-Shot Semantic Segmentation

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2025年第4期35卷 3185-3195页

作者： Zhang, Zicheng Ke, Wei Zhu, Yi Liang, Xiaodan Liu, Jianzhuang Ye, Qixiang Zhang, Tong Xi An Jiao Tong Univ Sch Software Engn Xian 710049 Peoples R China Huawei Technol Noahs Ark Lab Shenzhen 518129 Peoples R China Sun Yat Sen Univ Sch Intelligent Syst Engn Guangzhou 510275 Peoples R China Univ Chinese Acad Sci Sch Elect Elect & Commun Engn Beijing 100049 Peoples R China Ecole Polytech Fed Lausanne Image & Visual Representat Lab CH-1015 Lausanne Switzerland

The pre-trained vision-language model, exemplified by CLIP, advances zero-shot semantic segmentation by aligning visual features with class embeddings through a transformer decoder to generate semantic masks. Despite its effectiveness, prevailing methods within this paradigm encounter challenges, including overfitting on seen classes and small fragmentation in segmentation masks. To mitigate these issues, we propose a language-Driven Visual Consensus (LDVC) approach, fostering improved alignment of linguistic and visual information. Specifically, we leverage class embeddings as anchors due to their discrete and abstract nature, steering visual features toward class embeddings. Moreover, to achieve a more compact visual space, we introduce route attention into the transformer decoder to find visual consensus, thereby enhancing semantic consistency within the same object. Equipped with a vision-language prompting strategy, our approach significantly boosts the generalization capacity of segmentation models for unseen classes. Experimental results underscore the effectiveness of our approach, showcasing mIoU gains of 4.5% on the PASCAL VOC 2012 and 3.6% on the COCO-Stuff 164K for unseen classes compared with the state-of-the-art methods.

关键词： Visualization Tuning Decoding Transformers Semantic segmentation Adaptation models Semantics Streams Overfitting Biological system modeling vision-language model semantic segmentation zero-shot vision-language prompt tuning

来源：评论

学校读者我要写书评

暂无评论

Large language model-augmented learning for auto-delineation of treatment targets in head-and-neck cancer radiotherapy

引用

RADIOTHERAPY AND ONCOLOGY 2025年 205卷 110740-110740页

作者： Rajendran, Praveenbalaji Yang, Yong Niedermayr, Thomas R. Gensheimer, Michael Beadle, Beth Le, Quynh-Thu Xing, Lei Dai, Xianjin Stanford Univ Dept Radiat Oncol Stanford CA USA

Background and Purpose: Radiation therapy (RT) is highly effective, but its success depends on accurate, manual target delineation, which is time-consuming, labor-intensive, and prone to variability. Despite AI advancements in auto-contouring normal tissues, accurate RT target volume delineation remains challenging. This study presents Radformer, a novel visual language model that integrates text-rich clinical data with medical imaging for accurate automated RT target volume delineation. Materials and Methods: We developed Radformer, an innovative network that utilizes a hierarchical vision transformer as its backbone and integrates large language models (LLMs) to extract and embed clinical data in text-rich form. The model features a novel visual language attention module (VLAM) to combine visual and linguistic features, enabling language-aware visual encoding (LAVE). The Radformer was evaluated on a dataset of 2985 patients with head-and-neck cancer who underwent RT. Quantitative evaluations were performed utilizing metrics such as the Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95). Results: The Radformer demonstrated superior performance in segmenting RT target volumes compared to stateof-the-art models. On the head-and-neck cancer dataset, Radformer achieved a mean DSC of 0.76 f 0.09 versus 0.66 f 0.09, a mean IOU of 0.69 f 0.08 versus 0.59 f 0.07, and a mean HD95 of 7.82 f 6.87 mm versus 14.28 f 6.85 mm for gross tumor volume delineation, compared to the baseline 3D-UNETR. Conclusions: The Radformer model offers a clinically optimal means of RT target auto-delineation by integrating both imaging and clinical data through a visual language model. This approach improves the accuracy of RT target volume delineation, facilitating broader AI-assisted automation in RT treatment planning.

关键词： Auto-delineation Image segmentation Image-guided radiotherapy Deep learning Large language model vision-language model

来源：评论

学校读者我要写书评

暂无评论

Visual primitives as words: Alignment and interaction for compositional zero-shot

引用

PATTERN RECOGNITION 2025年 157卷

作者： Shuang, Feng Li, Jiahuan Huang, Qingbao Zhao, Wenye Xu, Dongsheng Han, Chao Cheng, Haonan Guangxi Univ Sch Elect Engn 100 East Daxue Rd Nanning 530004 Guangxi Peoples R China Guangxi Key Lab Intelligent Control & Maintenance 100 East Daxue Rd Nanning 530004 Guangxi Peoples R China Commun Univ China State Key Lab Media Convergence & Commun 1 Dingfuzhuang East St Beijing 100024 Peoples R China

Compositional Zero-Shot Learning (CZSL) aims to recognize seen and unseen attribute-object compositions. Recently, some researchers apply vision-language models to CZSL task. However, they only roughly match the image embedding and composition embedding on the image level, which can be a barrier to further improvement. With observation and analysis, we believe that a visual primitive is worth a word. To make full use of visual primitives to achieve fine-grained alignment and bridging modal gap, we propose VisPrompt for interacting visual primitives with sub-concepts in a prompt. Specifically, VisPrompt aligns the visual primitives (i.e., visual attribute and visual object) with the sub-concepts (i.e., text attribute and text object) at a finegrained level. It consists of two steps: (1) First, we extract the visual attribute embedding by an attribute extraction module, and the visual object embedding by an object extraction module;(2) Second, we design an attribute-wise prompt, an object-wise prompt, and a visual reconstructed prompt to be encoded, where a visual primitive plays the role of corresponding sub-concept to interact. Therefore, our model is capable of applying fine-grained alignment and bridging the gap between vision and text. Sufficient experiments on widely-used MIT-States, UT-Zappos, CGQA, and VAW-CZSL datasets show that our VisPromt achieves state-of-the-art on the core metric AUC.

关键词： Compositional zero-shot learning Attribute-object composition vision-language model Prompt tuning

来源：评论

学校读者我要写书评

暂无评论

AITtrack: Attention-Based Image-Text Alignment for Visual Tracking

引用

IEEE ACCESS 2025年 13卷 67095-67111页

作者： Alawode, Basit Javed, Sajid Khalifa Univ Sci Technol Dept Elect Engn & Comp Sci Abu Dhabi U Arab Emirates

vision-language models (VLMs) have recently advanced the Visual Object Tracking (VOT) performance. In VLMs, a vision encoder is employed to obtain visual representation, and a text encoder is employed to estimate the textual embeddings using natural language descriptions. By aligning the visual and textual representations, the VLMs achieve robust performance in complex and diverse tracking scenarios, efficiently handling dynamic target appearances such as motion blur, occlusion, fast motion, and similar object distractors. However, the input textual description in many existing VLM-based trackers incorporates class and semantics details without any contextual information. This is addressed in some recent VLM-based State Of The Art (SOTA) trackers by implicitly predicting some important attributes of the target object and encoding them as textual descriptions within the tracking paradigm. However, the SOTA methods neglect the contextual relationship among the predicted attributes. In this work, we propose an Attention-based Image-Text alignment Tracker (AITrack) for robust VOT tasks. AITrack simplifies the process of VLM-based tracking using attention-based visual and textual alignment modules. AITrack utilizes a region-of-interest (ROI) text-guided encoder that leverages existing pre-trained language models to implicitly extract and encode textual features and a simple image encoder to encode visual features. A simple alignment module is implemented to combine both encoded visual and textual features, thereby inherently exposing the semantic relationship between the template and search frames with their surroundings, providing rich encodings for improved tracking performance. We employ a simple decoder that takes past predictions as spatiotemporal clues to effectively model the target appearance changes without the need for complex customized post-processings and prediction heads. Extensive experiments are performed on six publicly available VOT benchmark datasets d

关键词： Target tracking Visualization Feature extraction Semantics Object tracking Correlation Filters Encoding Benchmark testing Head Visual object tracking multi-modal tracking vision-language model vision transformer autoregressive tracking

来源：评论

学校读者我要写书评

暂无评论

Fine-Tuning of CLIP in Few-Shot Scenarios via Supervised Contrastive Learning 7th

Fine-Tuning of CLIP in Few-Shot Scenarios via Supervised Con...

引用

7th Chinese Conference on Pattern Recognition and Computer vision

作者： Luo, Jing Wu, Guangxing Liu, Hongmei Wang, Ruixuan Sun Yat Sen Univ Sch Comp Sci & Engn Guangzhou Peoples R China Peng Cheng Lab Shenzhen Peoples R China MOE Key Lab Machine Intelligence & Adv Comp Guangzhou Peoples R China

ISBN: (纸本)9789819785018;9789819785025

Large-scale pretrained visual-language models like CLIP have proven highly effective in learning universal representations and achieved significant success across various downstream tasks. Recently, there has been increased interest in fine-tuning these large models with limited data. To alleviate the overfitting issue during fine-tuning, existing methods usually freeze the parameters of CLIP pretrained on large-scale datasets and use the features extracted from the CLIP model for downstream tasks. However, such fine-tuning strategy may limit the performance of these methods because semantic visual features specific for downstream tasks may not be well extracted based on the frozen feature extractor of CLIP. In this study, we propose an effective framework to fine-tune CLIP with few-shot samples and meanwhile alleviate the overfitting. In this framework, a visual adapter is embedded at the end of the CLIP's visual encoder to encourage the model to effectively extract semantic features relevant to the downstream task, and a supervised contrastive loss is introduced to alleviate the overfitting by guiding the optimisation to focus on the adapter. In addition, the multimodal feature alignment capability of CLIP is utilized to direct the adapted visual encoder in extracting class-relevant image features through textual prompts. Experimental evaluations on 11 datasets confirm the superior performance of the proposed approach.

关键词： Few-shot classification vision-language model Supervised contrastive learning

来源：评论

学校读者我要写书评

暂无评论

vision-language models Can Identify Distracted Driver Behavior From Naturalistic Videos

引用

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 2024年第9期25卷 11602-11616页

作者： Hasan, Md. Zahid Chen, Jiajing Wang, Jiyang Rahman, Mohammed Shaiqur Joshi, Ameya Velipasalar, Senem Hegde, Chinmay Sharma, Anuj Sarkar, Soumik Iowa State Univ Dept Elect & Comp Engn Ames IA 50010 USA Syracuse Univ Dept Elect Engn & Comp Sci Syracuse NY 13244 USA Iowa State Univ Dept Comp Sci Ames IA 50010 USA NYU Dept Elect & Comp Engn Brooklyn NY 11201 USA Iowa State Univ Dept Civil Construct & Environm Engn Ames IA 50010 USA Iowa State Univ Dept Mech Engn Ames IA 50010 USA

Recognizing the activities causing distraction in real-world driving scenarios is critical for ensuring the safety and reliability of both drivers and pedestrians on the roadways. Conventional computer vision techniques are typically data-intensive and require a large volume of annotated training data to detect and classify various distracted driving behaviors, thereby limiting their generalization ability, efficiency and scalability. We aim to develop a generalized framework that showcases robust performance with access to limited or no annotated training data. Recently, vision-language models have offered large-scale visual-textual pretraining that can be adapted to task-specific learning like distracted driving activity recognition. vision-language pretraining models like CLIP have shown significant promise in learning natural language-guided visual representations. This paper proposes a CLIP-based driver activity recognition approach that identifies driver distraction from naturalistic driving images and videos. CLIP's vision embedding offers zero-shot transfer and task-based finetuning, which can classify distracted activities from naturalistic driving video. Our results show that this framework offers state-of-the-art performance on zero-shot transfer, finetuning and video-based models for predicting the driver's state on four public datasets. We propose frame-based and video-based frameworks developed on top of the CLIP's visual representation for distracted driving detection and classification tasks and report the results. Our code is available at https://***/zahid-isu/DriveCLIP

关键词： Videos Vehicles Adaptation models Accidents Training Task analysis Data models Distracted driving computer vision CLIP vision-language model zero-shot transfer embedding

来源：评论

学校读者我要写书评

暂无评论

vision-language models for vision Tasks: A Survey

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024年第8期46卷 5625-5644页

作者： Zhang, Jingyi Huang, Jiaxing Jin, Sheng Lu, Shijian Nanyang Technol Univ Sch Comp Sci & Engn Singapore 639798 Singapore

Most visual recognition studies rely heavily on crowd-labelled data in deep neural networks (DNNs) training, and they usually train a DNN for each single visual recognition task, leading to a laborious and time-consuming visual recognition paradigm. To address the two challenges, vision-language models (VLMs) have been intensively investigated recently, which learns rich vision-language correlation from web-scale image-text pairs that are almost infinitely available on the Internet and enables zero-shot predictions on various visual recognition tasks with a single VLM. This paper provides a systematic review of visual language models for various visual recognition tasks, including: (1) the background that introduces the development of visual recognition paradigms;(2) the foundations of VLM that summarize the widely-adopted network architectures, pre-training objectives, and downstream tasks;(3) the widely-adopted datasets in VLM pre-training and evaluations;(4) the review and categorization of existing VLM pre-training methods, VLM transfer learning methods, and VLM knowledge distillation methods;(5) the benchmarking, analysis and discussion of the reviewed methods;(6) several research challenges and potential research directions that could be pursued in the future VLM studies for visual recognition.

关键词： Task analysis Visualization Training Deep learning Surveys Data models Predictive models Big Data big model deep learning deep neural network knowledge distillation object detection pre-training semantic segmentation transfer learning vision-language model visual recognition image classification

来源：评论

学校读者我要写书评

暂无评论

Semi-Automatic Labeling for Action Recognition by Diversity Preserving Sampling

Semi-Automatic Labeling for Action Recognition by Diversity ...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Ando, Ryuhei Shibata, Takashi Takahashi, Toru NEC Corporation Japan

ISBN: (纸本)9798350368741

Deep learning for action recognition is an important technology for understanding videos. However, collecting video training dataset for deep learning model with low cost while maintaining enough diversity is challenging. In this paper, we propose a semi-automatic labeling framework for action recognition by diversity-preserving sampling. The proposed framework utilizes a pre-trained vision-language model (VLM) to search through video clips to filter data that matches the text that describes the appropriate context for the target action. Since this simple approach by VLM tends to lack diversity, our framework is also equipped with diversity-preserving sampling that consists of two sampling strategies. One is confidence-based weighted sampling, which is based on action class confidence obtained from VLM, and the other is isolate-constraint-based weighted sampling, which samples points that are far apart in the text-image feature space. We conduct experiments to demonstrate that the proposed approach efficiently collects data with variations that could train a better action recognition model than the baseline. © 2025 IEEE.

关键词： action recognition labeling framework text-video retrieval vision-language model

来源：评论

学校读者我要写书评

暂无评论

Leveraging Visual Captions for Enhanced Zero-Shot HOI Detection

Leveraging Visual Captions for Enhanced Zero-Shot HOI Detect...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Zeng, Yanqing Mao, Yunyao Lu, Zhenbo Zhou, Wengang Li, Houqiang EEIS Department University of Science and Technology of China Hefei China Institute of Artificial Intelligence Hefei Comprehensive National Science Center Hefei China

ISBN: (纸本)9798350368741

Zero-shot Human-Object Interaction (HOI) detection aims to identify both seen and unseen HOI categories in an image. Most existing methods rely on semantic knowledge distilled from CLIP to find novel interactions but fail to fully exploit the powerful generalization ability of vision-language models, leading to impaired transferability. In this paper, we introduce a novel framework for zero-shot HOI detection. We first utilize vision-language models (VLMs) to generate visual captions from multiple perspectives, including humans, objects, and environments, to enhance interaction understanding. Then, we propose a multi-modal fusion encoder to fully leverage these visual captions. Additionally, to equip the HOI detector with a thorough consideration of contextual information in the image, we design a novel multi-branch HOI network that aggregates features at the instance, union, and global levels. Experiments on prevalent benchmarks demonstrate that our model achieves promising performance under a variety of zero-shot settings. The source codes are available at https://***/aqingcv/VC-HOI. © 2025 IEEE.

关键词： Human-object Interaction Multimodal fusion vision-language model Zero-shot

来源：评论

学校读者我要写书评

暂无评论

Attention Disentanglement for Semantic Diffusion modeling in Text-to-Image Generation

Attention Disentanglement for Semantic Diffusion Modeling in...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Yu, Hsiang-Chun Chien, Jen-Tsung Institute of Electrical and Computer Engineering National Yang Ming Chiao Tung University Hsinchu Taiwan

ISBN: (纸本)9798350368741

Text-to-image model has been recently improved to generate the semantically rich high-quality images by strengthening natural language processing via transformer in a stable diffusion process. However, the challenges are still remained in accurately rendering the objects, colors and compositions, and in precisely resolving the ambiguities in textual descriptions. This paper presents an attention disentanglment for semantic diffusion where the semantic consistency is enhanced for text-to-image generation. By utilizing the cross-attention maps in stable diffusion, this method is feasible to control the semantic features during generation process without the need for retraining. Additionally, a vision-language model is merged to implement this process to ensure that the generated images are closely aligned with the input prompts. This study further presents the attention disentanglement for semantic diffusion where the schemes of syntax parsing and information-theoretic learning are implemented to separate the semantic attributes and relations, thereby improving the accuracy and coherence of the generated images in the experiments. © 2025 IEEE.

关键词： attention disentanglement diffusion model mutual information vision-language model

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：