检索结果-内蒙古大学图书馆

Federated fine-grained prompts for vision-language models based on open-vocabulary object detection

APPLIED INTELLIGENCE 2025年第7期55卷 1-15页

作者： Li, Yu China Univ Petr East China Sch Comp Sci & Technol Qingdao 266580 Peoples R China

vision-language models can be used for open-vocabulary object detection. The existing methods suffer from low matching accuracy between prompt and image regions, as well as limited generalization capability as they adopt a data-centralized model training approach that ignores data heterogeneity. To alleviate these issues, we propose a federated fine-grained prompts learning method called FFPLearning, for open-vocabulary object detection using vision-language models. Specifically, FFPLearning quantifies the quality of proposals using pre-fused EoG (Energy of Gradient) and IoU (Intersection over Union) scores and organizes them into individual groups. Then learnable fine-grained prompts are trained to align the grouped region proposals in the feature space. A momentum update algorithm is designed to assess the quality of each participating client in the federated learning. Additionally, a Transformer-based feedback aggregation algorithm is designed to thoroughly leverage the semantic information from prompts and aggregate them based on the qualities of clients. Comprehensive evaluations on COCO and LVIS datasets demonstrate that FFPLearning is very effective, with +5.8 Novel AP50 and +3.3 APr improvements compared with existing state-of-the-art methods.

关键词： Federated learning vision-language model Open-vocabulary object detection

来源：评论

学校读者我要写书评

暂无评论

LVLM-EHub: A Comprehensive Evaluation Benchmark for Large vision-language models

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2025年第3期47卷 1877-1893页

作者： Xu, Peng Shao, Wenqi Zhang, Kaipeng Gao, Peng Liu, Shuo Lei, Meng Meng, Fanqing Huang, Siyuan Qiao, Yu Luo, Ping Shanghai AI Lab OpenGVLab Shanghai 200232 Peoples R China Univ Hong Kong Dept Comp Sci Hong Kong Peoples R China Shanghai AI Lab OpenGVLab Shanghai 200232 Peoples R China Peking Univ Beijing 100871 Peoples R China

Large vision-language models (LVLMs) have recently played a dominant role in multimodal vision-language learning. Despite the great success, it lacks a holistic evaluation of their efficacy. This paper presents a comprehensive evaluation of publicly available large multimodal models by building an LVLM evaluation Hub (LVLM-eHub). Our LVLM-eHub consists of 13 representative LVLMs such as InstructBLIP and LLaVA, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The former evaluates five categories of multimodal capabilities of LVLMs such as visual question answering and object hallucination on 42 in-domain text-related visual benchmarks, while the latter provides the user-level evaluation of LVLMs in an open-world question-answering scenario. The study investigates how specific features of LVLMs such as model configurations, modality alignment mechanisms, and training data affect the multimodal understanding. By conducting a comprehensive comparison of these features on quantitative and arena evaluation, our study uncovers several innovative findings, which establish a fundamental framework for the development and evaluation of innovative strategies aimed at enhancing multimodal techniques.

关键词： vision-language model multimodal evaluation large language model large language model multi-turn evaluation multi-turn evaluation multi-turn evaluation

来源：评论

学校读者我要写书评

暂无评论

RelVid: Relational Learning with vision-language models for Weakly Video Anomaly Detection

引用

SENSORS 2025年第7期25卷 2037-2037页

作者： Wang, Jingxin Li, Guohan Liu, Jiaqi Xu, Zhengyi Chen, Xinrong Wei, Jianming Chinese Acad Sci Shanghai Adv Res Inst Shanghai 201210 Peoples R China ShanghaiTech Univ Sch Informat Sci & Technol Shanghai 201210 Peoples R China Univ Chinese Acad Sci Sch Elect Elect & Commun Engn Beijing 100049 Peoples R China Fudan Univ Acad Engn & Technol Shanghai 200433 Peoples R China

Weakly supervised video anomaly detection aims to identify abnormal events in video sequences without requiring frame-level supervision, which is a challenging task in computer vision. Traditional methods typically rely on low-level visual features with weak supervision from a single backbone branch, which often struggles to capture the distinctive characteristics of different categories. This limitation reduces their adaptability to real-world scenarios. In real-world situations, the boundary between normal and abnormal events is often unclear and context-dependent. For example, running on a track may be considered normal, but running on a busy road could be deemed abnormal. To address these challenges, RelVid is introduced as a novel framework that improves anomaly detection by expanding the relative feature gap between classes extracted from a single backbone branch. The key innovation of RelVid lies in the integration of auxiliary tasks, which guide the model to learn more discriminative features, significantly boosting the model's performance. These auxiliary tasks-including text-based anomaly detection and feature reconstruction learning-act as additional supervision, helping the model capture subtle differences and anomalies that are often difficult to detect in weakly supervised settings. In addition, RelVid incorporates two other components, which include class activation feature learning for improved feature discrimination and a temporal attention module for capturing sequential dependencies. This approach enhances the model's robustness and accuracy, enabling it to better handle complex and ambiguous scenarios. Evaluations on two widely used benchmark datasets, UCF-Crime and XD-Violence, demonstrate the effectiveness of RelVid. Compared to state-of-the-art methods, RelVid achieves superior performance in both detection accuracy and robustness.

关键词： vision-language model Adapter weakly video anomaly detection feature learning

来源：评论

学校读者我要写书评

暂无评论

VaVLM: Toward Efficient Edge-Cloud Video Analytics With vision-language models

引用

IEEE TRANSACTIONS ON BROADCASTING 2025年

作者： Zhang, Yang Wang, Hanling Bai, Qing Liang, Haifeng Zhu, Peican Muntean, Gabriel-Miro Li, Qing Xian Technol Univ Sch Optoelect Engn Xian 710021 Shaanxi Peoples R China Pengcheng Lab Dept Adv Interdisciplinary Res Shenzhen 518055 Guangdong Peoples R China Tsinghua Univ Shenzhen Int Grad Sch Shenzhen 518055 Guangdong Peoples R China Northern Optoelect Co Ltd NORTHEO Dept Qual Safety Xian 710043 Shaanxi Peoples R China Northwestern Polytech Univ Sch Artificial Intelligence Opt & Elect Xian 710072 Shaanxi Peoples R China Dublin City Univ Sch Elect Engn Dublin 9 Ireland

The advancement of Large language models (LLMs) with vision capabilities in recent years has elevated video analytics applications to new heights. To address the limited computing and bandwidth resources on edge devices, edge-cloud collaborative video analytics has emerged as a promising paradigm. However, most existing edge-cloud video analytics systems are designed for traditional deep learning models (e.g., image classification and object detection), where each model handles a specific task. In this paper, we introduce VaVLM, a novel edge-cloud collaborative video analytics system tailored for vision-language models (VLMs), which can support multiple tasks using a single model. VaVLM aims to enhance the performance of VLM-powered video analytics systems in three key aspects. First, to reduce bandwidth consumption during video transmission, we propose a novel Region-of-Interest (RoI) generation mechanism based on the VLM's understanding of the task and scene. Second, to lower inference costs, we design a task-oriented inference trigger that processes only a subset of video frames using an optimized inference logic. Third, to improve inference accuracy, the model is augmented with additional information from both the environment and auxiliary analytics models during the inference stage. Extensive experiments on real-world datasets demonstrate that VaVLM achieves an 80.3% reduction in bandwidth consumption and an 89.5% reduction in computational cost compared to baseline methods.

关键词： Visual analytics Analytical models Computational modeling Image edge detection Cognition Bandwidth Artificial neural networks Streaming media Servers Object detection Video analytics vision-language model large language model edge computing

来源：评论

学校读者我要写书评

暂无评论

DPO: Discrete Prompt Optimization for vision-language models

引用

IEEE SIGNAL PROCESSING LETTERS 2025年 32卷 671-675页

作者： Liang, Nanhao Liu, Yong Chinese Acad Sci Hefei Inst Phys Sci Hefei 230031 Peoples R China Univ Sci & Technol China Hefei 230026 Peoples R China

In recent years, the emergence of large vision-language models (VLMs) has catalyzed the development of prompt learning, where networks are trained to enhance VLM performance by learning continuous prompts. However, traditional continuous prompt learning often struggles with challenges like overfitting to Base classes and a lack of interpretability due to the nature of prompt parameterization. To overcome these limitations, we introduce Discrete Prompt Optimization (DPO), a method that optimizes text prompts in discrete word-space. During training, scores are assigned to token embeddings, which are then used to select the most effective token sequence for the downstream task. DPO was tested across 11 diverse datasets, consistently outperforming baseline methods like CLIP and CoOp on Novel classes in most cases. This discrete approach not only reduces overfitting but also enhances transparency and model interpretability, enabling the learning of dataset-specific text prompts that are easily understandable.

关键词： Training Optimization Adaptation models Visualization Overfitting Vectors Vocabulary Signal processing algorithms Stochastic processes Standards Prompt learning vision-language model

来源：评论

学校读者我要写书评

暂无评论

Dual Modality Prompt Tuning for vision-language Pre-Trained model

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 2056-2068页

作者： Xing, Yinghui Wu, Qirui Cheng, De Zhang, Shizhou Liang, Guoqiang Wang, Peng Zhang, Yanning Northwestern Polytech Univ Sch Comp Sci Xian 710072 Peoples R China Northwestern Polytech Univ Shenzhen Res Dev Inst Shenzhen 518057 Peoples R China Xidian Univ Sch Telecommun Engn Xian 710071 Peoples R China

With the emergence of large pretrained vison-language models such as CLIP, transferable representations can be adapted to a wide range of downstream tasks via prompt tuning. Prompt tuning probes for beneficial information for downstream tasks from the general knowledge stored in the pretrained model. A recently proposed method named Context Optimization (CoOp) introduces a set of learnable vectors as text prompts from the language side. However, tuning the text prompt alone can only adjust the synthesized "classifier", while the computed visual features of the image encoder cannot be affected, thus leading to suboptimal solutions. In this article, we propose a novel dual-modality prompt tuning (DPT) paradigm through learning text and visual prompts simultaneously. To make the final image feature concentrate more on the target visual concept, a class-aware visual prompt tuning (CAVPT) scheme is further proposed in our DPT. In this scheme, the class-aware visual prompt is generated dynamically by performing the cross attention between text prompt features and image patch token embeddings to encode both the downstream task-related information and visual instance information. Extensive experimental results on 11 datasets demonstrate the effectiveness and generalization ability of the proposed method.

关键词： Visualization Tuning Task analysis Adaptation models Computational modeling Feature extraction Training Few-shot learning transfer learning image classification prompt tuning vision-language model

来源：评论

学校读者我要写书评

暂无评论

vision-language models for Design Concept Generation: An Actor–Critic Framework

引用

Journal of Mechanical Design 2025年第9期147卷 091402页

作者： Ghasemi, Parisa Moghaddam, Mohsen George W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta GA 30332

We introduce a novel actor-critic framework that utilizes vision-language models (VLMs) and large language models (LLMs) for design concept generation, particularly for producing a diverse array of innovative solutions to a given design problem. By leveraging the extensive data repositories and pattern recognition capabilities of these models, our framework achieves this goal through enabling iterative interactions between two VLM agents: an actor (i.e., concept generator) and a critic. The actor, a custom VLM (e.g., GPT-4) created using few-shot learning and fine-tuning techniques, generates initial design concepts that are improved iteratively based on guided feedback from the critic—a prompt-engineered LLM or a set of design-specific quantitative metrics. This process aims to optimize the generated concepts with respect to four metrics: novelty, feasibility, problem–solution relevancy, and variety. The framework incorporates both long-term and short-term memory models to examine how incorporating the history of interactions impacts decision-making and concept generation outcomes. We explored the efficacy of incorporating images alongside text in conveying design ideas within our actor–critic framework by experimenting with two mediums for the agents: vision language and language only. We extensively evaluated the framework through a case study using the AskNature dataset, comparing its performance against benchmarks such as GPT-4 and real-world biomimetic designs across various industrial examples. Our findings underscore the framework’s capability to iteratively refine and enhance the initial design concepts, achieving significant improvements across all metrics. We conclude by discussing the implications of the proposed framework for various design domains, along with its limitations and several directions for future research in this domain.

关键词： large language model Design agent-based design creativity and concept generation Large language models feasibility machine learning artificial intelligence design concept generation novelty Feedback conceptual design data-driven design vision-language model bioinspired design creativity

来源：评论

学校读者我要写书评

暂无评论

Federated Prototype Guided Adaption for vision-language models

Federated Prototype Guided Adaption for Vision-Language Mode...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Liu, Youchao Huang, Dingjiang School of Data Science and Engineering East China Normal University Shanghai China

ISBN: (纸本)9798350368741

Federated Learning (FL) is a new pivotal paradigm for decentralized training on heterogeneous data. Recently fine-tuning of vision-language models (VLMs) has been extended to the federated setting to improve overall performance. Unfortunately, in this case, FL still faces two critical challenges that hinder its actual performance: data distribution heterogeneity and high resource costs brought by large VLMs. In this paper, we introduce FedPGA, a prototype-guided method, for achieving performance improvements in the federated setting for VLMs. Concretely, we design a prototype-based adapter for the vision-language model, CLIP. The lightweight adapter updates the prior knowledge encoded in CLIP to enhance its adaption capability further and avoid the effects of data distribution heterogeneity in the federated setting. Simultaneously, small-scale operations can mitigate the computational and communication burden caused by large VLMs. Our comprehensive empirical evaluations of nine diverse image classification datasets show that our method is superior to existing FL methods under VLMs. © 2025 IEEE.

关键词： federated learning few-shot classification prototype-based adapter vision-language model

来源：评论

学校读者我要写书评

暂无评论

Improving Anomaly Scene Recognition with Large vision-language models 18th

Improving Anomaly Scene Recognition with Large Vision-Langu...

引用

18th International Conference on Wireless Artificial Intelligent Computing Systems and Applications, WASA 2024

作者： Liu, Cheng Long, Xianlei Li, Yan Chen, Chao Gu, Fuqiang Yuan, Songyu Zhang, Chunlong Chongqing University Chongqing401331 China Macquarie University SydneyNSW2109 Australia Zhejiang Lab Hangzhou311121 China

ISBN: (纸本)9783031714696

vision-based anomaly scene recognition is important for plenty of applications such as surveillance and security. An efficient way to achieve anomaly scene recognition is to use image-based methods. However, the accuracy achieved by existing image-based methods is relatively low. In this work, we introduce AnoSRe, a novel method designed to accurately recognize anomaly scenes from images. AnoSRe takes advantage of the pre-trained large vision-language model to improve the accuracy of popular deep learning networks for anomaly scene recognition. Experiments conducted on two datasets show that the proposed method outperforms state-of-the-art methods, and improves the accuracy by from 5% to 23% on UCF-Crime-Image and IDSR datasets. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

关键词： Anomaly detection Anomaly scene recognition Large language model vision-language model

来源：评论

学校读者我要写书评

暂无评论

LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in vision-language models

LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Express...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Wang, Jingyi Ju, Jianzhong Luan, Jian Deng, Zhidong Department of Computer Science and Technology BNRist THUAI Tsinghua University Beijing China Xiaomi AI Lab Beijing China

ISBN: (纸本)9798350368741

Recent advances in large vision-language models (LVLMs) typically employ vision encoders based on the vision Transformer (ViT) architecture. The division of the images into patches by ViT results in a fragmented perception, thereby hindering the visual understanding capabilities of LVLMs. In this paper, we propose an innovative enhancement to address this limitation by introducing a Scene Graph Expression (SGE) module in LVLMs. This module extracts and structurally expresses the complex semantic information within images, thereby improving the foundational perception and understanding abilities of LVLMs. Extensive experiments demonstrate that integrating our SGE module significantly enhances the LVLM's performance in vision-language tasks, indicating its effectiveness in preserving intricate semantic details and facilitating better visual understanding. © 2025 IEEE.

关键词： Large language model Scene Graph vision-language model

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：