检索结果-内蒙古大学图书馆

DGPrompt: Dual-guidance prompts generation for vision-language models

NEURAL NETWORKS 2025年 188卷

作者： Zheng, Tai Chen, Zhen-Duo Zhang, Zi-Chao Ma, Zhen-Xiang Zhao, Li-Jun Zhang, Chong-Yu Luo, Xin Xu, Xin-Shun Shandong Univ Sch Software 1500 Shunhua Rd Jinan 250101 Peoples R China

Introducing learnable prompts into CLIP and fine-tuning them have demonstrated excellent performance across many downstream tasks. However, existing methods have insufficient interaction between modalities and neglect the importance of hierarchical contextual information, leading to ineffective alignment in both the visual and textual representation spaces. Additionally, CLIP is highly sensitive to prompts, making learnable prompts prone to overfitting on seen classes, which results in the forgetting of general knowledge of CLIP and severely impair generalization ability on unseen classes. To address these issues, we propose an original Dual-Guidance Prompts Generation (DGPrompt) method that promotes alignment between visual and textual spaces while ensuring the continuous retention of general knowledge. The main ideas of DGPrompt are as follows: 1) The extraction of image and text embeddings are guided mutually by generating visual and textual prompts, making full use of complementary information from both modalities to align visual and textual spaces. 2) The prompt-tuning process is restrained by a retention module, reducing the forgetting of general knowledge. Extensive experiments conducted in settings of base-to-new class generalization and few-shot learning demonstrate the superiority of the proposed method. Compared with the baseline method CLIP and the state-of-the-art method MaPLe, DGPrompt exhibits favorable performance and achieves an absolute gain of 7.84% and 0.99% on overall harmonic mean, averaged over 11 diverse image recognition datasets.

关键词： Prompt-tuning vision-language model Modalities alignment Generalization capability

来源：评论

学校读者我要写书评

暂无评论

DPO: Discrete Prompt Optimization for vision-language models

引用

IEEE SIGNAL PROCESSING LETTERS 2025年 32卷 671-675页

作者： Liang, Nanhao Liu, Yong Chinese Acad Sci Hefei Inst Phys Sci Hefei 230031 Peoples R China Univ Sci & Technol China Hefei 230026 Peoples R China

In recent years, the emergence of large vision-language models (VLMs) has catalyzed the development of prompt learning, where networks are trained to enhance VLM performance by learning continuous prompts. However, traditional continuous prompt learning often struggles with challenges like overfitting to Base classes and a lack of interpretability due to the nature of prompt parameterization. To overcome these limitations, we introduce Discrete Prompt Optimization (DPO), a method that optimizes text prompts in discrete word-space. During training, scores are assigned to token embeddings, which are then used to select the most effective token sequence for the downstream task. DPO was tested across 11 diverse datasets, consistently outperforming baseline methods like CLIP and CoOp on Novel classes in most cases. This discrete approach not only reduces overfitting but also enhances transparency and model interpretability, enabling the learning of dataset-specific text prompts that are easily understandable.

关键词： Training Optimization Adaptation models Visualization Overfitting Vectors Vocabulary Signal processing algorithms Stochastic processes Standards Prompt learning vision-language model

来源：评论

学校读者我要写书评

暂无评论

Dual Modality Prompt Tuning for vision-language Pre-Trained model

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 2056-2068页

作者： Xing, Yinghui Wu, Qirui Cheng, De Zhang, Shizhou Liang, Guoqiang Wang, Peng Zhang, Yanning Northwestern Polytech Univ Sch Comp Sci Xian 710072 Peoples R China Northwestern Polytech Univ Shenzhen Res Dev Inst Shenzhen 518057 Peoples R China Xidian Univ Sch Telecommun Engn Xian 710071 Peoples R China

With the emergence of large pretrained vison-language models such as CLIP, transferable representations can be adapted to a wide range of downstream tasks via prompt tuning. Prompt tuning probes for beneficial information for downstream tasks from the general knowledge stored in the pretrained model. A recently proposed method named Context Optimization (CoOp) introduces a set of learnable vectors as text prompts from the language side. However, tuning the text prompt alone can only adjust the synthesized "classifier", while the computed visual features of the image encoder cannot be affected, thus leading to suboptimal solutions. In this article, we propose a novel dual-modality prompt tuning (DPT) paradigm through learning text and visual prompts simultaneously. To make the final image feature concentrate more on the target visual concept, a class-aware visual prompt tuning (CAVPT) scheme is further proposed in our DPT. In this scheme, the class-aware visual prompt is generated dynamically by performing the cross attention between text prompt features and image patch token embeddings to encode both the downstream task-related information and visual instance information. Extensive experimental results on 11 datasets demonstrate the effectiveness and generalization ability of the proposed method.

关键词： Visualization Tuning Task analysis Adaptation models Computational modeling Feature extraction Training Few-shot learning transfer learning image classification prompt tuning vision-language model

来源：评论

学校读者我要写书评

暂无评论

vision-language models for Design Concept Generation: An Actor–Critic Framework

引用

Journal of Mechanical Design 2025年第9期147卷 091402页

作者： Ghasemi, Parisa Moghaddam, Mohsen George W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta GA 30332

We introduce a novel actor-critic framework that utilizes vision-language models (VLMs) and large language models (LLMs) for design concept generation, particularly for producing a diverse array of innovative solutions to a given design problem. By leveraging the extensive data repositories and pattern recognition capabilities of these models, our framework achieves this goal through enabling iterative interactions between two VLM agents: an actor (i.e., concept generator) and a critic. The actor, a custom VLM (e.g., GPT-4) created using few-shot learning and fine-tuning techniques, generates initial design concepts that are improved iteratively based on guided feedback from the critic—a prompt-engineered LLM or a set of design-specific quantitative metrics. This process aims to optimize the generated concepts with respect to four metrics: novelty, feasibility, problem–solution relevancy, and variety. The framework incorporates both long-term and short-term memory models to examine how incorporating the history of interactions impacts decision-making and concept generation outcomes. We explored the efficacy of incorporating images alongside text in conveying design ideas within our actor–critic framework by experimenting with two mediums for the agents: vision language and language only. We extensively evaluated the framework through a case study using the AskNature dataset, comparing its performance against benchmarks such as GPT-4 and real-world biomimetic designs across various industrial examples. Our findings underscore the framework’s capability to iteratively refine and enhance the initial design concepts, achieving significant improvements across all metrics. We conclude by discussing the implications of the proposed framework for various design domains, along with its limitations and several directions for future research in this domain.

关键词： large language model Design agent-based design creativity and concept generation Large language models feasibility machine learning artificial intelligence design concept generation novelty Feedback conceptual design data-driven design vision-language model bioinspired design creativity

来源：评论

学校读者我要写书评

暂无评论

vision-language models in medical image analysis: From simple fusion to general large models

引用

INFORMATION FUSION 2025年 118卷

作者： Li, Xiang Li, Like Jiang, Yuchen Wang, Hao Qiao, Xinyu Feng, Ting Luo, Hao Zhao, Yong Northeastern Univ Coll Informat Sci & Engn Shenyang 110819 Peoples R China Hebei Key Lab Micronano Precis Opt Sensing & Measu Qinhuangdao 066004 Peoples R China Harbin Inst Technol Dept Control Sci & Engn Harbin 150001 Peoples R China State Key Lab Synthet Automat Proc Ind Shenyang 110819 Peoples R China

vision-language model (VLM) is a kind of multi-modality deep learning model that aims to fuse visual information with language information to enhance the understanding and analysis of visual content. VLM was originally used to integrate multi-modality information and improve task accuracy. Then, VLM was further developed in combination with zero-shot and few-shot learning to solve the problem of insufficient medical labels. At present, it is the technical basis of the popular medical general large model. Its role is no longer limited to simple information fusion. This paper makes a comprehensive review for the development and application of VLM-based medical image analysis technology. Specifically, this paper first introduces the basic principle and explains the pre-training and fine-tuning framework. Then, the research progress of medical image classification, segmentation, report generation, question answering, image generation, large model and other application scenarios is introduced. This paper also summarizes seven main characteristics of medical image VLM, and analyzes the specific embodiment of these characteristics in each task. Finally, the challenges, potential solutions and future directions in this field are discussed. VLM is still in a rapid development in the field of medical image analysis, and a continuously updated repository of papers and code has been built, it is available at https://***/XiangQA-Q/VLM-in-MIA.

关键词： vision-language model Modality fusion Medical image analysis Large model

来源：评论

学校读者我要写书评

暂无评论

Improving Anomaly Scene Recognition with Large vision-language models 18th

Improving Anomaly Scene Recognition with Large Vision-Langu...

引用

18th International Conference on Wireless Artificial Intelligent Computing Systems and Applications, WASA 2024

作者： Liu, Cheng Long, Xianlei Li, Yan Chen, Chao Gu, Fuqiang Yuan, Songyu Zhang, Chunlong Chongqing University Chongqing401331 China Macquarie University SydneyNSW2109 Australia Zhejiang Lab Hangzhou311121 China

ISBN: (纸本)9783031714696

vision-based anomaly scene recognition is important for plenty of applications such as surveillance and security. An efficient way to achieve anomaly scene recognition is to use image-based methods. However, the accuracy achieved by existing image-based methods is relatively low. In this work, we introduce AnoSRe, a novel method designed to accurately recognize anomaly scenes from images. AnoSRe takes advantage of the pre-trained large vision-language model to improve the accuracy of popular deep learning networks for anomaly scene recognition. Experiments conducted on two datasets show that the proposed method outperforms state-of-the-art methods, and improves the accuracy by from 5% to 23% on UCF-Crime-Image and IDSR datasets. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： Anomaly detection Anomaly scene recognition Large language model vision-language model

来源：评论

学校读者我要写书评

暂无评论

Federated Prototype Guided Adaption for vision-language models

Federated Prototype Guided Adaption for Vision-Language Mode...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Liu, Youchao Huang, Dingjiang School of Data Science and Engineering East China Normal University Shanghai China

ISBN: (纸本)9798350368741

Federated Learning (FL) is a new pivotal paradigm for decentralized training on heterogeneous data. Recently fine-tuning of vision-language models (VLMs) has been extended to the federated setting to improve overall performance. Unfortunately, in this case, FL still faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large VLMs. In this paper, we introduce FedPGA, a prototype-guided method, for achieving performance improvements in the federated setting for VLMs. Concretely, we design a prototype-based adapter for the vision-language model, CLIP. The lightweight adapter updates the prior knowledge encoded in CLIP to enhance its adaption capability further and avoid the effects of data distribution heterogeneity in the federated setting. Simultaneously, small-scale operations can mitigate the computational and communication burden caused by large VLMs. Our comprehensive empirical evaluations of nine diverse image classification datasets show that our method is superior to existing FL methods under VLMs. © 2025 IEEE.

关键词： federated learning few-shot classification prototype-based adapter vision-language model

来源：评论

学校读者我要写书评

暂无评论

LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in vision-language models

LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Express...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Wang, Jingyi Ju, Jianzhong Luan, Jian Deng, Zhidong Department of Computer Science and Technology BNRist THUAI Tsinghua University Beijing China Xiaomi AI Lab Beijing China

ISBN: (纸本)9798350368741

Recent advances in large vision-language models (LVLMs) typically employ vision encoders based on the vision Transformer (ViT) architecture. The division of the images into patches by ViT results in a fragmented perception, thereby hindering the visual understanding capabilities of LVLMs. In this paper, we propose an innovative enhancement to address this limitation by introducing a Scene Graph Expression (SGE) module in LVLMs. This module extracts and structurally expresses the complex semantic information within images, thereby improving the foundational perception and understanding abilities of LVLMs. Extensive experiments demonstrate that integrating our SGE module significantly enhances the LVLM's performance in vision-language tasks, indicating its effectiveness in preserving intricate semantic details and facilitating better visual understanding. © 2025 IEEE.

关键词： Large language model Scene Graph vision-language model

来源：评论

学校读者我要写书评

暂无评论

vision-language pre-training for graph-based handwritten mathematical expression recognition

引用

PATTERN RECOGNITION 2025年 162卷

作者： Guo, Hong-Yu Wang, Chuang Yin, Fei Li, Xiao-Hui Liu, Cheng-Li Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100049 Peoples R China Chinese Acad Sci Inst Automat State Key Lab Multimodal Artificial Intelligence S Beijing 100190 Peoples R China

vision-language pre-training models have shown promise in improving various downstream tasks. However, handwritten mathematical expression recognition (HMER), as atypical structured learning problem, can hardly benefit from existing pre-training methods due to the presence of multiple symbols and complicated structural relationships, as well as the scarcity of paired data. To overcome these problems, we propose a V ision-language P re-training paradigm for G raph-based HMER (VLPG), utilizing unpaired mathematical expression images and LaTeX labels. Our HMER model is built upon a graph parsing method with superior explainability, which is enhanced by the proposed graph-structure aware transformer decoder. Based on this framework, the symbol localization pretext task and language modeling task are employed for vision-language pre-training. First, we make use of unlabeled mathematical symbol images to pre-train the visual feature extractor through the localization pretext task, improving the symbol localization and discrimination ability. Second, the structure understanding module is pre-trained using LaTeX corpora through language modeling task, which promotes the model's context comprehension ability. The pre-trained model is fine-tuned and aligned on the downstream HMER task using benchmark datasets. Experiments on public datasets demonstrate that the pre-training paradigm significantly improves the mathematical expression recognition performance. Our VLPG achieves state-of-the-art performance on standard CROHME datasets and comparable performance on the HME100K dataset, highlighting the effectiveness and superiority of the proposed model. We released our codes at https://***/guohy17/VLPG.

关键词： Pre-training Pretext task language modeling vision-language model Handwritten mathematical expression recognition

来源：评论

学校读者我要写书评

暂无评论

Generalizable Prompt Learning via Gradient Constrained Sharpness-Aware Minimization

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2025年 27卷 1100-1113页

作者： Liu, Liangchen Wang, Nannan Zhou, Dawei Liu, Decheng Yang, Xi Gao, Xinbo Liu, Tongliang Xidian Univ Sch Telecommun Engn State Key Lab Integrated Serv Networks Xian 710071 Peoples R China Xidian Univ Sch Cyber Engn State Key Lab Integrated Serv Networks Xian 710071 Peoples R China Chongqing Univ Posts & Telecommun Chongqing Key Lab Image Cognit Chongqing 400065 Peoples R China Univ Sydney Fac Engn Sydney AI Ctr Sch Comp Sci Darlington NSW 2008 Australia

This paper targets a novel trade-off problem in generalizable prompt learning for vision-language models (VLM), i.e., improving the performance on unseen classes while maintaining the performance on seen classes. Comparing with existing generalizable methods that neglect the seen classes degradation, the setting of this problem is stricter and fits more closely with practical applications. To solve this problem, we start from the optimization perspective, and leverage the relationship between loss landscape geometry and model generalization ability. By analyzing the loss landscapes of the state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based method, we conclude that the trade-off performance correlates to both loss value and loss sharpness, while each of them is indispensable. However, we find the optimizing gradient of existing methods cannot maintain high relevance to both loss value and loss sharpness during optimization, which severely affects their trade-off performance. To this end, we propose a novel SAM-based method for prompt learning, denoted as Gradient Constrained Sharpness-aware Context Optimization (GCSCoOp), to dynamically constrain the optimizing gradient, thus achieving above two-fold optimization objective simultaneously. Extensive experiments verify the effectiveness of GCSCoOp in the trade-off problem.

关键词： Optimization Minimization Visualization Training Electronic mail Degradation Vectors Telecommunications Intserv networks Geometry Parameter-efficient fine-tuning domain generalization prompt learning vision-language model sharpness-aware minimization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：