检索结果-内蒙古大学图书馆

vision-language models in medical image analysis: From simple fusion to general large models

INFORMATION FUSION 2025年 118卷

作者： Li, Xiang Li, Like Jiang, Yuchen Wang, Hao Qiao, Xinyu Feng, Ting Luo, Hao Zhao, Yong Northeastern Univ Coll Informat Sci & Engn Shenyang 110819 Peoples R China Hebei Key Lab Micronano Precis Opt Sensing & Measu Qinhuangdao 066004 Peoples R China Harbin Inst Technol Dept Control Sci & Engn Harbin 150001 Peoples R China State Key Lab Synthet Automat Proc Ind Shenyang 110819 Peoples R China

vision-language model (VLM) is a kind of multi-modality deep learning model that aims to fuse visual information with language information to enhance the understanding and analysis of visual content. VLM was originally used to integrate multi-modality information and improve task accuracy. Then, VLM was further developed in combination with zero-shot and few-shot learning to solve the problem of insufficient medical labels. At present, it is the technical basis of the popular medical general large model. Its role is no longer limited to simple information fusion. This paper makes a comprehensive review for the development and application of VLM-based medical image analysis technology. Specifically, this paper first introduces the basic principle and explains the pre-training and fine-tuning framework. Then, the research progress of medical image classification, segmentation, report generation, question answering, image generation, large model and other application scenarios is introduced. This paper also summarizes seven main characteristics of medical image VLM, and analyzes the specific embodiment of these characteristics in each task. Finally, the challenges, potential solutions and future directions in this field are discussed. VLM is still in a rapid development in the field of medical image analysis, and a continuously updated repository of papers and code has been built, it is available at https://***/XiangQA-Q/VLM-in-MIA.

关键词： vision-language model Modality fusion Medical image analysis Large model

来源：评论

学校读者我要写书评

暂无评论

vision-language models Can Identify Distracted Driver Behavior From Naturalistic Videos

引用

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS 2024年第9期25卷 11602-11616页

作者： Hasan, Md. Zahid Chen, Jiajing Wang, Jiyang Rahman, Mohammed Shaiqur Joshi, Ameya Velipasalar, Senem Hegde, Chinmay Sharma, Anuj Sarkar, Soumik Iowa State Univ Dept Elect & Comp Engn Ames IA 50010 USA Syracuse Univ Dept Elect Engn & Comp Sci Syracuse NY 13244 USA Iowa State Univ Dept Comp Sci Ames IA 50010 USA NYU Dept Elect & Comp Engn Brooklyn NY 11201 USA Iowa State Univ Dept Civil Construct & Environm Engn Ames IA 50010 USA Iowa State Univ Dept Mech Engn Ames IA 50010 USA

Recognizing the activities causing distraction in real-world driving scenarios is critical for ensuring the safety and reliability of both drivers and pedestrians on the roadways. Conventional computer vision techniques are typically data-intensive and require a large volume of annotated training data to detect and classify various distracted driving behaviors, thereby limiting their generalization ability, efficiency and scalability. We aim to develop a generalized framework that showcases robust performance with access to limited or no annotated training data. Recently, vision-language models have offered large-scale visual-textual pretraining that can be adapted to task-specific learning like distracted driving activity recognition. vision-language pretraining models like CLIP have shown significant promise in learning natural language-guided visual representations. This paper proposes a CLIP-based driver activity recognition approach that identifies driver distraction from naturalistic driving images and videos. CLIP's vision embedding offers zero-shot transfer and task-based finetuning, which can classify distracted activities from naturalistic driving video. Our results show that this framework offers state-of-the-art performance on zero-shot transfer, finetuning and video-based models for predicting the driver's state on four public datasets. We propose frame-based and video-based frameworks developed on top of the CLIP's visual representation for distracted driving detection and classification tasks and report the results. Our code is available at https://***/zahid-isu/DriveCLIP

关键词： Videos Vehicles Adaptation models Accidents Training Task analysis Data models Distracted driving computer vision CLIP vision-language model zero-shot transfer embedding

来源：评论

学校读者我要写书评

暂无评论

UMPA: Unified multi-modal prompt with adapter for vision-language models

引用

MULTIMEDIA SYSTEMS 2025年第2期31卷 1-11页

作者： Jin, Zhengwei Wei, Yun Univ Shanghai Sci & Technol Sch Opt Elect & Comp Engn Shanghai 200093 Peoples R China

Large-scale multi-modal pretraining model, such as CLIP, has shown remarkable generalization in vision-language tasks. However, the transfer of large models to downstream tasks requires large-scale computing resources, so adapter is proposed to realize fine-tuning for downstream tasks. As input text, prompt can guide CLIP learn the correspondence between images and texts when performing specific tasks. The selection of prompt template is very sensitive. In this paper, we propose Unified Multi-modal Prompt with Adapter For vision-language models (UMPA) based on CLIP, for parameter-efficient fine-tuning (PEFT). Learnable prompt design can improve the adaption ability of model. Adapter can realize lightweight model fine-tuning. With applying learnable prompt and adapter both on vision and language branches while the entire pre-trained parameters freeze, the model improve alignment between the vision and language representations. The model can learn separate visual representations across different stages to extract stage-wise feature relationships progressively, achieve comprehensive learning of rich visual information. Considering the additional learning modules, we apply image attention masking to prevent overfitting in downstream tasks. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness and strong few-shot generalization ability of our approach.

关键词： vision-language model CLIP Parameter-efficient fine-tuning Image classification

来源：评论

学校读者我要写书评

暂无评论

Towards label-free defect detection in additive manufacturing via dual-classifier semi-supervised learning for vision-language models

引用

JOURNAL OF INTELLIGENT MANUFACTURING 2025年 1-16页

作者： Wang, Kang Liu, Lanqing Xu, Cheng Zou, Jing Lin, Haoneng Fang, Naiyu Jiang, Jingchao Nanyang Technol Univ Singapore Singapore Hong Kong Polytech Univ Hung Hom Kowloon Hong Kong Peoples R China Univ Exeter Exeter England

Complex components can now be fabricated in innovative ways thanks to additive manufacturing (AM) technology, but it also presents a severe challenge in the detection of defects, primarily due to extensive labeling efforts of defect samples. In order to address the labeling problem in AM defect identification, this work suggests a unique strategy called the Dual-Classifier Semi-supervised Learning (DCSL) method. DCSL reduces the requirement for intensive tagging and improves detection accuracy by utilizing both labeled and unlabeled data. Two distinct classifiers, namely the one-hot classifier and the semantic classifier, are designed to train the defect class labels from different perspectives. The one-hot classifier adopts a one-hot encoding of labels, treating each class independently from the visual perspective, while the semantic classifier employs a distributed representation of labels, grouping potentially similar classes from the natural language view. The incorporation of dual classifiers transforms the proposed DCSL into a vision-language model, leveraging the semantic classifier to improve pseudo-labeling quality by identifying semantic relationships among class labels. Extensive tests on the publicly accessible AM defect dataset confirm that DCSL is more efficacious than state-of-the-art techniques for AM defect identification at the moment. The findings show that our DCSL is designed to enhance image discrimination skills towards label-free defect identification by training classifiers with different data perspectives in a synergistic manner. Given its innovative approach to defect detection, which can improve the effectiveness and precision of quality control and ultimately aid in the broad adoption of intelligent manufacturing across numerous industries, this study has the potential to herald in a new era of embodied intelligence in the current manufacturing system.

关键词： Defect detection Additive manufacturing Semi-supervised learning vision-language model Label free Embodied intelligence

来源：评论

学校读者我要写书评

暂无评论

DGPrompt: Dual-guidance prompts generation for vision-language models

引用

NEURAL NETWORKS 2025年 188卷

作者： Zheng, Tai Chen, Zhen-Duo Zhang, Zi-Chao Ma, Zhen-Xiang Zhao, Li-Jun Zhang, Chong-Yu Luo, Xin Xu, Xin-Shun Shandong Univ Sch Software 1500 Shunhua Rd Jinan 250101 Peoples R China

Introducing learnable prompts into CLIP and fine-tuning them have demonstrated excellent performance across many downstream tasks. However, existing methods have insufficient interaction between modalities and neglect the importance of hierarchical contextual information, leading to ineffective alignment in both the visual and textual representation spaces. Additionally, CLIP is highly sensitive to prompts, making learnable prompts prone to overfitting on seen classes, which results in the forgetting of general knowledge of CLIP and severely impair generalization ability on unseen classes. To address these issues, we propose an original Dual-Guidance Prompts Generation (DGPrompt) method that promotes alignment between visual and textual spaces while ensuring the continuous retention of general knowledge. The main ideas of DGPrompt are as follows: 1) The extraction of image and text embeddings are guided mutually by generating visual and textual prompts, making full use of complementary information from both modalities to align visual and textual spaces. 2) The prompt-tuning process is restrained by a retention module, reducing the forgetting of general knowledge. Extensive experiments conducted in settings of base-to-new class generalization and few-shot learning demonstrate the superiority of the proposed method. Compared with the baseline method CLIP and the state-of-the-art method MaPLe, DGPrompt exhibits favorable performance and achieves an absolute gain of 7.84% and 0.99% on overall harmonic mean, averaged over 11 diverse image recognition datasets.

关键词： Prompt-tuning vision-language model Modalities alignment Generalization capability

来源：评论

学校读者我要写书评

暂无评论

Federated fine-grained prompts for vision-language models based on open-vocabulary object detection

引用

APPLIED INTELLIGENCE 2025年第7期55卷 1-15页

作者： Li, Yu China Univ Petr East China Sch Comp Sci & Technol Qingdao 266580 Peoples R China

vision-language models can be used for open-vocabulary object detection. The existing methods suffer from low matching accuracy between prompt and image regions, as well as limited generalization capability as they adopt a data-centralized model training approach that ignores data heterogeneity. To alleviate these issues, we propose a federated fine-grained prompts learning method called FFPLearning, for open-vocabulary object detection using vision-language models. Specifically, FFPLearning quantifies the quality of proposals using pre-fused EoG (Energy of Gradient) and IoU (Intersection over Union) scores and organizes them into individual groups. Then learnable fine-grained prompts are trained to align the grouped region proposals in the feature space. A momentum update algorithm is designed to assess the quality of each participating client in the federated learning. Additionally, a Transformer-based feedback aggregation algorithm is designed to thoroughly leverage the semantic information from prompts and aggregate them based on the qualities of clients. Comprehensive evaluations on COCO and LVIS datasets demonstrate that FFPLearning is very effective, with +5.8 Novel AP50 and +3.3 APr improvements compared with existing state-of-the-art methods.

关键词： Federated learning vision-language model Open-vocabulary object detection

来源：评论

学校读者我要写书评

暂无评论

LVLM-EHub: A Comprehensive Evaluation Benchmark for Large vision-language models

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2025年第3期47卷 1877-1893页

作者： Xu, Peng Shao, Wenqi Zhang, Kaipeng Gao, Peng Liu, Shuo Lei, Meng Meng, Fanqing Huang, Siyuan Qiao, Yu Luo, Ping Shanghai AI Lab OpenGVLab Shanghai 200232 Peoples R China Univ Hong Kong Dept Comp Sci Hong Kong Peoples R China Shanghai AI Lab OpenGVLab Shanghai 200232 Peoples R China Peking Univ Beijing 100871 Peoples R China

Large vision-language models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building an LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of 13 representative LVLMs such as InstructBLIP and LLaVA, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates five categories of multimodal capabilities of LVLMs such as visual question answering and object hallucination on 42 in-domain text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study investigates how specific features of LVLMs such as model configurations, modality alignment mechanisms, and training data affect the multimodal understanding. By conducting a comprehensive comparison of these features on quantitative and arena evaluation, our study uncovers several innovative findings, which establish a fundamental framework for the development and evaluation of innovative strategies aimed at enhancing multimodal techniques.

关键词： vision-language model multimodal evaluation large language model large language model multi-turn evaluation multi-turn evaluation multi-turn evaluation

来源：评论

学校读者我要写书评

暂无评论

CLIP-Adapter: Better vision-language models with Feature Adapters

引用

INTERNATIONAL JOURNAL OF COMPUTER vision 2024年第2期132卷 581-595页

作者： Gao, Peng Geng, Shijie Zhang, Renrui Ma, Teli Fang, Rongyao Zhang, Yongfeng Li, Hongsheng Qiao, Yu Shanghai AI Lab Shanghai Peoples R China Rutgers State Univ New Brunswick NJ USA Chinese Univ Hong Kong Hong Kong Peoples R China

Large-scale contrastive vision-language pretraining has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in Radford et al. (International conference on machine learning, PMLR, 2021) to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions. To avoid non-trivial prompt engineering, context optimization (Zhou et al. in Int J Comput Vis 130(9):2337-2348, 2022) has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples. In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning. While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pretrained features. As a consequence, CLIP-Adapter is able to outperform context optimization while maintaining a simple design. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.

关键词： Feature adapter vision-language model Few-shot learning Open-vocabulary

来源：评论

学校读者我要写书评

暂无评论

Multi-Modal Attribute Prompting for vision-language models

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2024年第11期34卷 11579-11591页

作者： Liu, Xin Wu, Jiamin Yang, Wenfei Zhou, Xu Zhang, Tianzhu Univ Sci & Technol China Sch Informat Sci & Technol Hefei 230027 Peoples R China Sangfor Technol Inc Shenzhen 518000 Peoples R China

Pre-trained vision-language models (VLMs), like CLIP, exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. This limitation hinders the model's ability to perceive fine-grained visual details and restricts its generalization ability to a broader range of unseen classes. To address this issue, we propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment. The proposed MAP enjoys several merits. First, we introduce learnable visual attribute prompts enhanced by textual attribute semantics to adaptively capture visual attributes for images from unknown categories, boosting fine-grained visual perception capabilities for CLIP. Second, the proposed attribute-level alignment complements the global alignment to enhance the robustness of cross-modal alignment for open-vocabulary objects. To our knowledge, this is the first work to establish cross-modal attribute-level alignment for CLIP-based few-shot adaptation. Extensive experimental results on 11 datasets demonstrate that our method performs favorably against state-of-the-art approaches.

关键词： Few-shot classification prompt learning vision-language model attribute

来源：评论

学校读者我要写书评

暂无评论

VaVLM: Toward Efficient Edge-Cloud Video Analytics With vision-language models

引用

IEEE TRANSACTIONS ON BROADCASTING 2025年

作者： Zhang, Yang Wang, Hanling Bai, Qing Liang, Haifeng Zhu, Peican Muntean, Gabriel-Miro Li, Qing Xian Technol Univ Sch Optoelect Engn Xian 710021 Shaanxi Peoples R China Pengcheng Lab Dept Adv Interdisciplinary Res Shenzhen 518055 Guangdong Peoples R China Tsinghua Univ Shenzhen Int Grad Sch Shenzhen 518055 Guangdong Peoples R China Northern Optoelect Co Ltd NORTHEO Dept Qual Safety Xian 710043 Shaanxi Peoples R China Northwestern Polytech Univ Sch Artificial Intelligence Opt & Elect Xian 710072 Shaanxi Peoples R China Dublin City Univ Sch Elect Engn Dublin 9 Ireland

The advancement of Large language models (LLMs) with vision capabilities in recent years has elevated video analytics applications to new heights. To address the limited computing and bandwidth resources on edge devices, edge-cloud collaborative video analytics has emerged as a promising paradigm. However, most existing edge-cloud video analytics systems are designed for traditional deep learning models (e.g., image classification and object detection), where each model handles a specific task. In this paper, we introduce VaVLM, a novel edge-cloud collaborative video analytics system tailored for vision-language models (VLMs), which can support multiple tasks using a single model. VaVLM aims to enhance the performance of VLM-powered video analytics systems in three key aspects. First, to reduce bandwidth consumption during video transmission, we propose a novel Region-of-Interest (RoI) generation mechanism based on the VLM's understanding of the task and scene. Second, to lower inference costs, we design a task-oriented inference trigger that processes only a subset of video frames using an optimized inference logic. Third, to improve inference accuracy, the model is augmented with additional information from both the environment and auxiliary analytics models during the inference stage. Extensive experiments on real-world datasets demonstrate that VaVLM achieves an 80.3% reduction in bandwidth consumption and an 89.5% reduction in computational cost compared to baseline methods.

关键词： Visual analytics Analytical models Computational modeling Image edge detection Cognition Bandwidth Artificial neural networks Streaming media Servers Object detection Video analytics vision-language model large language model edge computing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：