检索结果-内蒙古大学图书馆

Federated fine-grained prompts for vision-language models based on open-vocabulary object detection

APPLIED INTELLIGENCE 2025年第7期55卷 1-15页

作者： Li, Yu China Univ Petr East China Sch Comp Sci & Technol Qingdao 266580 Peoples R China

Vision-language models can be used for open-vocabulary object detection. The existing methods suffer from low matching accuracy between prompt and image regions, as well as limited generalization capability as they adopt a data-centralized model training approach that ignores data heterogeneity. To alleviate these issues, we propose a federated fine-grained prompts learning method called FFPLearning, for open-vocabulary object detection using vision-language models. Specifically, FFPLearning quantifies the quality of proposals using pre-fused EoG (Energy of Gradient) and IoU (Intersection over Union) scores and organizes them into individual groups. Then learnable fine-grained prompts are trained to align the grouped region proposals in the feature space. A momentum update algorithm is designed to assess the quality of each participating client in the federated learning. Additionally, a Transformer-based feedback aggregation algorithm is designed to thoroughly leverage the semantic information from prompts and aggregate them based on the qualities of clients. Comprehensive evaluations on COCO and LVIS datasets demonstrate that FFPLearning is very effective, with +5.8 Novel AP50 and +3.3 APr improvements compared with existing state-of-the-art methods.

关键词： Federated learning Vision-language model open-vocabulary object detection

来源：评论

学校读者我要写书评

暂无评论

Reframing: Detector-Specific Prompt Tuning for Enhancing open-vocabulary object detection 19th

Reframing: Detector-Specific Prompt Tuning for Enhancing Ope...

引用

19th International Conference on Hybrid Artificial Intelligence Systems

作者： Avshalumov, Mikhail Volovikova, Zoya Yudin, Dmitry Panov, Alexandr Moscow Inst Phys & Technol Dolgoprudnyi 141701 Russia AIRI Moscow 121170 Russia Russian Acad Sci Fed Res Ctr Comp Sci & Control Moscow Russia

ISBN: (纸本)9783031741852;9783031741869

object detection in images using natural language prompts has recently become popular due to the extensive development of open-vocabulary perception tasks. We propose an enhancement strategy for open-vocabulary object detection models through a fine-tuning approach named Reframing. The cornerstone of our method is transforming a user's query into a prompt optimally aligned with the requirements of a specific detection model. This transformation is facilitated by integrating an additional Large Language Model, which undergoes further training. We propose online and offline modes of Reframing, employing deep learning techniques and feedback from the detection models to refine its query transformation process. Our methodology has been applied to various detection models, showcasing its capability to significantly improve prediction accuracy. Developed code and dataset are available at https://***/ZoyaV/reframing.

关键词： open-vocabulary object detection Image recognition Prompt tuning Large language model

来源：评论

学校读者我要写书评

暂无评论

Boosting open-vocabulary object detection Performance via Class-Agnostic Pseudo-Labels and MultiModal Hybrid Knowledge

Boosting Open-Vocabulary Object Detection Performance via Cl...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Chen, Ziyang Liu, Dongqin Dai, Jiao Hu, Songlin Institute of Information Engineering Chinese Academy of Sciences Beijing China School of Cyber Security University of Chinese Academy of Sciences Beijing China

ISBN: (纸本)9798350368741

open-vocabulary object detection (OVD) is a significant task identifying objects from categories not included in the training set. Our comprehensive analysis reveals two main issues with existing OVD models: poor generalization of localization network to novel categories and poor quality of class embedding impacting accuracy. We propose two solutions: Localization Network Enhancement based on Class-Agnostic Pseudo-Labels (LNE-CAPL) and Class Embedding Enhancement based on MultiModal Hybrid Knowledge (CEE-MMHK). These methods can be used offline to significantly improve the existing OVD models without affecting their training and inference efficiency. Extensive experiments confirm their effectiveness and universality. Code and models will soon be open-sourced. © 2025 IEEE.

关键词： class embedding enhancement localization network enhancement open-vocabulary object detection

来源：评论

学校读者我要写书评

暂无评论

Enhancing open-vocabulary object detection through region-word and region-vision matching

引用

MULTIMEDIA SYSTEMS 2025年第3期31卷 1-15页

作者： Chen, Yi Wang, Chong Li, Zhehao Lin, Sunqi Xiang, Jinhui Li, Yuqi Qian, Jiangbo Ningbo Univ Fac Elect Engn & Comp Sci Ningbo 315000 Zhejiang Peoples R China

open-vocabulary object detection (OVOD) aims to detect novel object categories beyond the training set. Existing OVOD methods have made encouraging progress by leveraging large-scale image-caption pairs and pre-trained vision-language models (VLMs). However, two main limitations exhibit: (1) The potential category-specific concepts in global captions are not fully utilized, resulting in a lack of fine-grained semantic guidance for the detector. (2) The compositional structure of multiple concepts naturally existing in image-caption pairs as represented by VLMs remains insufficiently explored, limiting the model's ability to generalize to novel category concepts. To address these limitations, we propose a novel framework called Region-Word-Vision Matching (RWVM) that integrates two core modules: a Region-Word Matching (RWM) module and a Region-Vision Matching (RVM) module. Our key insight is to simultaneously guide textual and visual knowledge alignment with region features to strengthen the model's understanding of complex visual scenes. Specifically, the RWM module guides fine-grained semantic aggregation by fusing local region-word matching with global image-caption matching. The RVM module leverages VLMs to capture the compositional structure of single and multiple object concepts, directly enhancing the detector's ability to learn novel category concepts. Additionally, we demonstrate that the RVM module outperforms embeddings extracted from full language models using only simplified region embeddings. Extensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the average precision (AP) for novel categories on COCO and LVIS datasets.

关键词： open-vocabulary object detection Region-word matching Vision-language models Multi-modal training

来源：评论

学校读者我要写书评

暂无评论

open-vocabulary object detection by Novel-Class Feature Perception Enhancement 20th

Open-Vocabulary Object Detection by Novel-Class Feature Perc...

引用

20th International Conference on Intelligent Computing (ICIC)

作者： Hui, Kanghua Cai, Xianqiao Zhang, Zhi Huang, Rui Liu, Qing Civil Aviat Univ China Coll Comp Sci & Technol Tianjin 300300 Peoples R China Qingdao Civil Aviat Cares Co Ltd Qingdao 266000 Peoples R China

ISBN: (纸本)9789819755905;9789819755912

open-vocabulary object detection (OVD) tasks have been advanced by enhancing the regional representation capability of the Constrastive Language-Image Pre-training (CLIP). However, as CLIP is trained with image-text pairs, lacking precise object location information, merely enhancing the regional representation capability of the model offers limited improvement in accurately locating novel classes. To address this shortfall, we employ Image Masking (IM) through uniform distribution-based non-repetitive random sampling to remove a large amount of redundant and distracting image blocks, enabling the model to better perceive the semantic information of novel classes. Furthermore, we design an Improved Detailed Feature (IDF) extraction network, which uses original image features in a newbranch to supplement detail information and to reconstruct multi-scale features. This mitigates the loss of detail information and the decrease in base-class accuracy caused by discarding a large number of image blocks, and further enhances the detection capability for novel classes. Our method achieves 36.2% mAPr on OV-LVIS and 45.5% AP50novel on OV-COCO, respectively. The abundant experiments have demonstrated the superior performance of our method over other compared methods in OVD task.

关键词： open-vocabulary object detection Image Masking Semantic Information Improved Detailed Feature

来源：评论

学校读者我要写书评

暂无评论

Fine-Grained VisualText Prompt-Driven Self-Training for open-vocabulary object detection

引用

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024年第11期35卷 16277-16287页

作者： Long, Yanxin Han, Jianhua Huang, Runhui Xu, Hang Zhu, Yi Xu, Chunjing Liang, Xiaodan Sun Yat Sen Univ Shenzhen Sch Intelligent Syst Engn Shenzhen 518107 Peoples R China Huawei Noahs Ark Lab Shanghai 201206 Peoples R China DarkMatter AI Res Guangzhou 511458 Peoples R China

Inspired by the success of vision-language methods (VLMs) in zero-shot classification, recent works attempt to extend this line of work into object detection by leveraging the localization ability of pretrained VLMs and generating pseudolabels for unseen classes in a self-training manner. However, since the current VLMs are usually pretrained with aligning sentence embedding with global image embedding, the direct use of them lacks fine-grained alignment for object instances, which is the core of detection. In this article, we propose a simple but effective fine-grained visual-text prompt-driven self-training paradigm for open-vocabulary detection (VTP-OVD) that introduces a fine-grained visual-text prompt adapting stage to enhance the current self-training paradigm with a more powerful fine-grained alignment. During the adapting stage, we enable VLM to obtain fine-grained alignment using learnable text prompts to resolve an auxiliary dense pixelwise prediction task. Furthermore, we propose a visual prompt module to provide the prior task information (i.e., the categories need to be predicted) for the vision branch to better adapt the pretrained VLM to the downstream tasks. Experiments show that our method achieves the state-of-the-art performance for open-vocabulary object detection, e.g., 31.5% mAP on unseen classes of COCO.

关键词： Task analysis object detection Visualization Tuning Pipelines Adaptation models Training open-vocabulary object detection prompt learning self-training vision-language

来源：评论

学校读者我要写书评

暂无评论

Multi-modal Prompts with Feature Decoupling for open-vocabulary object detection 2nd

Multi-modal Prompts with Feature Decoupling for Open-Vocabul...

引用

2nd International Workshop on Generalizing from Limited Resources in the open World (GLOW)

作者： Wang, Duorui Zhao, Xiaowei Beihang Univ State Key Lab Complex & Crit Software Environm Beijing 100191 Peoples R China

ISBN: (纸本)9789819761241;9789819761258

open-vocabulary object detection aims to acquire the ability to recognize novel categories through text description using data of limited categories for training. The Prompt serves as a template to assist in the construction of textual descriptions for categories. With the development of open-vocabulary object detection, multi-modal prompts with better performance have emerged. However, existing multi-modal prompts fail to align the context and object components across different modalities during the construction. To address the issue, we propose an open-vocabulary object detection framework based on multi-modal prompts with feature decoupling. The framework consists of two modules, the construction of Multi-modal Prompts with Feature Decoupling (MPFD) and the visual Region Expansion (RE). During prompts constructing, the MPFD decouples the object and context components from the visual embeddings and then performs multi-modal fusion with the corresponding parts of the text embeddings respectively. The RE incorporates additional context information into the visual embeddings to enhance the discriminative ability of the prompts. Sufficient experiments have demonstrated that feature decoupling multi-modal prompts can effectively improve the performance of open-vocabulary object detection models.

关键词： feature decoupling multi-modal prompts open-vocabulary object detection region expansion

来源：评论

学校读者我要写书评

暂无评论

Taming Self-Training for open-vocabulary object detection

Taming Self-Training for Open-Vocabulary Object Detection

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Zhao, Shiyu Schulter, Samuel Zhao, Long Zhang, Zhixing Kumar, Vijay B. G. Suh, Yumin Chandraker, Manmohan Metaxas, Dimitris N. Rutgers State Univ New Brunswick NJ 08901 USA NEC Labs Amer Princeton NJ USA Google Res Mountain View CA USA Univ Calif San Diego San Diego CA USA

ISBN: (纸本)9798350353006

Recent studies have shown promising performance in open-vocabulary object detection (OVD) by utilizing pseudo labels (PLs) from pretrained vision and language models (VLMs). However, teacher-student self-training, a powerful and widely used paradigm to leverage PLs, is rarely explored for OVD. This work identifies two challenges of using self-training in OVD: noisy PLs from VLMs and frequent distribution changes of PLs. To address these challenges, we propose SAS-Det that tames self-training for OVD from two key perspectives. First, we present a split-and-fusion (SAF) head that splits a standard detection into an open-branch and a closed-branch. This design can reduce noisy supervision from pseudo boxes. Moreover, the two branches learn complementary knowledge from different training data, significantly enhancing performance when fused together. Second, in our view, unlike in closed-set tasks, the PL distributions in OVD are solely determined by the teacher model. We introduce a periodic update strategy to decrease the number of updates to the teacher, thereby decreasing the frequency of changes in PL distributions, which stabilizes the training process. Extensive experiments demonstrate SAS-Det is both efficient and effective. SAS-Det outperforms recent models of the same scale by a clear margin and achieves 37.4 AP50 and 29.1 APr on novel categories of the COCO and LVIS benchmarks, respectively. Code is available at https://***/xiaofeng94/SAS-Det.

关键词： open-vocabulary object detection pseudo label self-training vision and language

来源：评论

学校读者我要写书评

暂无评论

Retrieval-Augmented open-vocabulary object detection

Retrieval-Augmented Open-Vocabulary Object Detection

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Kim, Jooyeon Cho, Eulrang Kim, Sehyung Kim, Hyunwoo J. Korea Univ Dept Comp Sci & Engn Seoul South Korea Samsung Res Mountain View CA USA

ISBN: (纸本)9798350353006

open-vocabulary object detection (OVD) has been studied with Vision-Language Models (VLMs) to detect novel objects beyond the pre-trained categories. Previous approaches improve the generalization ability to expand the knowledge of the detector, using 'positive' pseudo-labels with additional 'class' names, e.g., sock, iPod, and alligator. To extend the previous methods in two aspects, we propose Retrieval-Augmented Losses and visual Features ( RALF). Our method retrieves related 'negative' classes and augments loss functions. Also, visual features are augmented with 'verbalized concepts' of classes, e.g., worn on the feet, handheld music player, and sharp teeth. Specifically, RALF consists of two modules: Retrieval Aug-mented Losses ( RAL) and Retrieval-Augmented visual Features (RAF). RAL constitutes two losses reflecting the semantic similarity with negative vocabularies. In addition, RAF augments visual features with the verbalized concepts from a large language model (LLM). Our experiments demonstrate the effectiveness of RALF on COCO and LVIS benchmark datasets. We achieve improvement up to 3.4 box APN on novel categories of the COCO dataset and 3.6 mask APr gains on the LVIS dataset. Code is available at https://***/mlvlab/RALF.

关键词： Computer Vision object detection open-vocabulary object detection Retrieval-Augmentation

来源：评论

学校读者我要写书评

暂无评论

Exploring Region-Word Alignment in Built-in Detector for open-vocabulary object detection

Exploring Region-Word Alignment in Built-in Detector for Ope...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Zhang, Heng Zhao, Qiuyu Zheng, Linyu Zeng, Hao Ge, Zhiwei Li, Tianhao Xu, Sulong JD Com Hefei Peoples R China

ISBN: (纸本)9798350353006

open-vocabulary object detection aims to detect novel categories that are independent from the base categories used during training. Most modern methods adhere to the paradigm of learning vision-language space from a large-scale multi-modal corpus and subsequently transferring the acquired knowledge to off-the-shelf detectors like Faster-RCNN. However, information attenuation or destruction may occur during the process of knowledge transfer due to the domain gap, hampering the generalization ability on novel categories. To mitigate this predicament, in this paper, we present a novel framework named BIND, standing for Bulit-IN Detector, to eliminate the need for module replacement or knowledge transfer to off-the-shelf detectors. Specifically, we design a two-stage training framework with an Encoder-Decoder structure. In the first stage, an image-text dual encoder is trained to learn region-word alignment from a corpus of image-text pairs. In the second stage, a DETR-style decoder is trained to perform detection on annotated object detection datasets. In contrast to conventional manually designed non-adaptive anchors, which generate numerous redundant proposals, we develop an anchor proposal network that generates anchor proposals with high likelihood based on candidates adaptively, thereby substantially improving detection efficiency. Experimental results on two public benchmarks, COCO and LVIS, demonstrate that our method stands as a state-of-the-art approach for open-vocabulary object detection.

关键词： open-vocabulary object detection Region-word alignment

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：