检索结果-内蒙古大学图书馆

Robotic environmental state recognition with pre-trained vision-language models and black-box optimization

ADVANCED ROBOTICS 2024年第18期38卷 1255-1264页

作者： Kawaharazuka, Kento Obinata, Yoshiki Kanazawa, Naoaki Okada, Kei Inaba, Masayuki Univ Tokyo Grad Sch Informat Sci & Technol Dept Mechanoinformat Bunkyo Ku Tokyo Japan

In order for robots to autonomously navigate and operate in diverse environments, it is essential for them to recognize the state of their environment. On the other hand, the environmental state recognition has traditionally involved distinct methods tailored to each state to be recognized. In this study, we perform a unified environmental state recognition for robots through the spoken language with pre-trained large-scale vision-language models. We apply Visual Question Answering and Image-to-Text Retrieval, which are tasks of vision-language models. We show that with our method, it is possible to recognize not only whether a room door is open/closed, but also whether a transparent door is open/closed and whether water is running in a sink, without training neural networks or manual programing. In addition, the recognition accuracy can be improved by selecting appropriate texts from the set of prepared texts based on black-box optimization. For each state recognition, only the text set and its weighting need to be changed, eliminating the need to prepare multiple different models and programs, and facilitating the management of source code and computer resources. We experimentally demonstrate the effectiveness of our method and apply it to the recognition behavior on a mobile robot, Fetch.

关键词： vision-language model black-box optimization environmental state recognition

来源：评论

学校读者我要写书评

暂无评论

Multi-task prompt tuning with soft context sharing for vision-language models

引用

NEUROCOMPUTING 2024年 603卷

作者： Ding, Kun Wang, Ying Liu, Pengzhang Yu, Qiang Zhang, Haojian Xiang, Shiming Pan, Chunhong Chinese Acad Sci Inst Automat State Key Lab Multimodal Artificial Intelligence S Beijing Peoples R China Chinese Acad Sci Inst Automat Engn Lab Intelligent Ind Vis Beijing Peoples R China Chinese Acad Sci Inst Automat Res Ctr Aerosp Informat Beijing Peoples R China JD Com Beijing Peoples R China

vision-language models have recently shown great potential on many tasks in computer vision. Meanwhile, prior work demonstrates prompt tuning designed for vision-language models could acquire superior performance on few-shot image recognition compared to linear probe, a strong baseline. In practice, many few-shot tasks are inherently correlated, particularly within specialized domains. However, such information is overlooked previously. Inspired by the fact that modeling task relationship by multi-task learning can usually boost performance, we propose a novel method SoftCPT (Soft Context Sharing for Prompt Tuning) to tune pre-trained vision-language models on multiple target few-shot tasks jointly. Specifically, we design a task- shared meta network to generate prompt context for each task using task name together with a learnable task context as input. The parameters of this meta network as well as the task context are tuned on the joint training set of all tasks. As such, the prompt context of all tasks will be shared in a soft manner. Extensive experiments across four multi-task few-shot datasets covering 44 tasks and 1593 categories demonstrate that SoftCPT significantly outperforms single-task prompt tuning methods, highlighting the effectiveness of multi-task learning for vision-language prompt tuning.

关键词： Prompt tuning vision-language model Multi-task learning Few-shot recognition

来源：评论

学校读者我要写书评

暂无评论

Rectify representation bias in vision-language models for long-tailed recognition

引用

NEURAL NETWORKS 2024年 172卷 106134页

作者： Li, Bo Yao, Yongqiang Tan, Jingru Gong, Ruihao Lu, Jianwei Luo, Ye Tongji Univ 4800 Caoan Rd Shanghai 201804 Peoples R China Sensetime Res 1900 Hongmei Rd Shanghai 201103 Peoples R China Cent South Univ 932 South Lushan Rd Changsha 410083 Hunan Peoples R China Shanghai Univ Tradit Chinese Med 530 Lingling Rd Shanghai 201203 Peoples R China

Natural data typically exhibits a long-tailed distribution, presenting great challenges for recognition tasks. Due to the extreme scarcity of training instances, tail classes often show inferior performance. In this paper, we investigate the problem within the trendy visual-language (VL) framework and find that the performance bottleneck mainly arises from the recognition confusion between tail classes and their highly correlated head classes. Building upon this observation, unlike previous research primarily emphasizing class frequency in addressing long-tailed issues, we take a novel perspective by incorporating a crucial additional factor namely class correlation. Specifically, we model the representation learning procedure for each sample as two parts, i.e., a special part that learns the unique properties of its own class and a common part that learns shared characteristics among classes. By analysis, we discover that the learning process of common representation is easily biased toward head classes. Because of the bias, the network may lean towards the biased common representation as classification criteria, rather than prioritizing the crucial information encapsulated within the specific representation, ultimately leading to recognition confusion. To solve the problem, based on the VL framework, we introduce the rectification contrastive term (ReCT) to rectify the representation bias, according to semantic hints and training status. Extensive experiments on three widely-used long-tailed datasets demonstrate the effectiveness of ReCT. On iNaturalist2018, it achieves an overall accuracy of 75.4%, surpassing the baseline by 3.6 points in a ResNet-50 visual backbone.

关键词： Long-tailed recognition vision-language model Representation bias

来源：评论

学校读者我要写书评

暂无评论

Integrating vision-language models for Accelerated High-Throughput Nutrition Screening

引用

ADVANCED SCIENCE 2024年第34期11卷 e2403578页

作者： Ma, Peihua Wu, Yixin Yu, Ning Jia, Xiaoxue He, Yiyang Zhang, Yang Backes, Michael Wang, Qin Wei, Cheng-, I Univ Maryland Coll Agr & Nat Resources Dept Nutr & Food Sci College Pk MD 20742 USA CISPA Helmholtz Ctr Informat Secur D-66123 Saarbrucken Germany Netflix Eyeline Studios Los Angeles CA 90028 USA

Addressing the critical need for swift and precise nutritional profiling in healthcare and in food industry, this study pioneers the integration of vision-language models (VLMs) with chemical analysis techniques. A cutting-edge VLM is unveiled, utilizing the expansive UMDFood-90k database, to significantly improve the speed and accuracy of nutrient estimation processes. Demonstrating a macro-AUCROC of 0.921 for lipid quantification, the model exhibits less than 10% variance compared to traditional chemical analyses for over 82% of the analyzed food items. This innovative approach not only accelerates nutritional screening by 36.9% when tested amongst students but also sets a new benchmark in the precision of nutritional data compilation. This research marks a substantial leap forward in food science, employing a blend of advanced computational models and chemical validation to offer a rapid, high-throughput solution for nutritional analysis. This study integrates vision-language models with chemical analysis techniques to revolutionize nutritional profiling. Utilizing the extensive UMDFood-90k database, the novel UMDFood-VL model significantly enhances the accuracy and speed of nutrient estimation, reducing experiment time by 36.9% and achieving a macro-AUCROC of 0.921 for lipid quantification. This approach sets a new standard in precision nutrition and food science. image

关键词： food analysis high-throughput screening machine learning precision nutrition vision-language model

来源：评论

学校读者我要写书评

暂无评论

vision-language pre-training for graph-based handwritten mathematical expression recognition

引用

PATTERN RECOGNITION 2025年 162卷

作者： Guo, Hong-Yu Wang, Chuang Yin, Fei Li, Xiao-Hui Liu, Cheng-Li Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100049 Peoples R China Chinese Acad Sci Inst Automat State Key Lab Multimodal Artificial Intelligence S Beijing 100190 Peoples R China

vision-language pre-training models have shown promise in improving various downstream tasks. However, handwritten mathematical expression recognition (HMER), as atypical structured learning problem, can hardly benefit from existing pre-training methods due to the presence of multiple symbols and complicated structural relationships, as well as the scarcity of paired data. To overcome these problems, we propose a V ision-language P re-training paradigm for G raph-based HMER (VLPG), utilizing unpaired mathematical expression images and LaTeX labels. Our HMER model is built upon a graph parsing method with superior explainability, which is enhanced by the proposed graph-structure aware transformer decoder. Based on this framework, the symbol localization pretext task and language modeling task are employed for vision-language pre-training. First, we make use of unlabeled mathematical symbol images to pre-train the visual feature extractor through the localization pretext task, improving the symbol localization and discrimination ability. Second, the structure understanding module is pre-trained using LaTeX corpora through language modeling task, which promotes the model's context comprehension ability. The pre-trained model is fine-tuned and aligned on the downstream HMER task using benchmark datasets. Experiments on public datasets demonstrate that the pre-training paradigm significantly improves the mathematical expression recognition performance. Our VLPG achieves state-of-the-art performance on standard CROHME datasets and comparable performance on the HME100K dataset, highlighting the effectiveness and superiority of the proposed model. We released our codes at https://***/guohy17/VLPG.

关键词： Pre-training Pretext task language modeling vision-language model Handwritten mathematical expression recognition

来源：评论

学校读者我要写书评

暂无评论

Attention head purification: A new perspective to harness CLIP for domain generalization

引用

IMAGE AND vision COMPUTING 2025年 157卷

作者： Wang, Yingfan Kang, Guoliang Beihang Univ 37 Xueyuan Rd Beijing 100191 Peoples R China

Domain Generalization (DG) aims to learn a model from multiple source domains to achieve satisfactory performance on unseen target domains. Recent works introduce CLIP to DG tasks due to its superior image-text alignment and zeros-shot performance. Previous methods either utilize full fine-tuning or prompt-learning paradigms to harness CLIP for DG tasks. Those works focus on avoiding catastrophic forgetting of the original knowledge encoded in CLIP but ignore that the knowledge encoded in CLIP in nature may contain domain-specific cues that constrain its domain generalization performance. In this paper, we propose a new perspective to harness CLIP for DG, i.e., attention head purification. We observe that different attention heads may encode different properties of an image and selecting heads appropriately may yield remarkable performance improvement across domains. Based on such observations, we purify the attention heads of CLIP from two levels, including task-level purification and domain-level purification. For task-level purification, we design head-aware LoRA to make each head more adapted to the task we considered. For domain-level purification, we perform head selection via a simple gating strategy. We utilize MMD loss to encourage masked head features to be more domain-invariant to emphasize more generalizable properties/heads. During training, we jointly perform task-level purification and domain-level purification. We conduct experiments on various representative DG benchmarks. Though simple, extensive experiments demonstrate that our method performs favorably against previous state-of-the-arts.

关键词： Multi-source domain generalization vision-language model Low-rank adaptation

来源：评论

学校读者我要写书评

暂无评论

Generalizable Prompt Learning via Gradient Constrained Sharpness-Aware Minimization

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2025年 27卷 1100-1113页

作者： Liu, Liangchen Wang, Nannan Zhou, Dawei Liu, Decheng Yang, Xi Gao, Xinbo Liu, Tongliang Xidian Univ Sch Telecommun Engn State Key Lab Integrated Serv Networks Xian 710071 Peoples R China Xidian Univ Sch Cyber Engn State Key Lab Integrated Serv Networks Xian 710071 Peoples R China Chongqing Univ Posts & Telecommun Chongqing Key Lab Image Cognit Chongqing 400065 Peoples R China Univ Sydney Fac Engn Sydney AI Ctr Sch Comp Sci Darlington NSW 2008 Australia

This paper targets a novel trade-off problem in generalizable prompt learning for vision-language models (VLM), i.e., improving the performance on unseen classes while maintaining the performance on seen classes. Comparing with existing generalizable methods that neglect the seen classes degradation, the setting of this problem is stricter and fits more closely with practical applications. To solve this problem, we start from the optimization perspective, and leverage the relationship between loss landscape geometry and model generalization ability. By analyzing the loss landscapes of the state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based method, we conclude that the trade-off performance correlates to both loss value and loss sharpness, while each of them is indispensable. However, we find the optimizing gradient of existing methods cannot maintain high relevance to both loss value and loss sharpness during optimization, which severely affects their trade-off performance. To this end, we propose a novel SAM-based method for prompt learning, denoted as Gradient Constrained Sharpness-aware Context Optimization (GCSCoOp), to dynamically constrain the optimizing gradient, thus achieving above two-fold optimization objective simultaneously. Extensive experiments verify the effectiveness of GCSCoOp in the trade-off problem.

关键词： Optimization Minimization Visualization Training Electronic mail Degradation Vectors Telecommunications Intserv networks Geometry Parameter-efficient fine-tuning domain generalization prompt learning vision-language model sharpness-aware minimization

来源：评论

学校读者我要写书评

暂无评论

IndVisSGG: VLM-based scene graph generation for industrial spatial intelligence

引用

ADVANCED ENGINEERING INFORMATICS 2025年 65卷

作者： Wang, Zuoxu Yan, Zhijie Li, Shufei Liu, Jihong Beihang Univ Sch Mech Engn & Automat Beijing Peoples R China Nanyang Technol Univ Ctr Adv Robot Technol Innovat Singapore Singapore Huazhong Univ Sci & Technol State Key Lab Digital Mfg Equipment & Technol Wuhan Peoples R China

Industrial spatial intelligence enables robots and machine tools to understand environmental settings and their relationships, allowing them to manipulate target components. A crucial aspect of this process is scene graph generation (SGG). Previous research on SGG primarily focuses on detection and panoptic segmentation of objects, followed by the prediction of their pairwise relationships. However, these approaches struggle with generalization and transferability when encountering new scenarios. To tackle this problem, we propose the Industrial Visual Scene Graph Generation (IndVisSGG) method, which parses spatial and interactive relationships between objects in temporal industrial settings. This approach leverages the capabilities of vision-language models (VLMs) to generate scene graphs quickly and accurately without any additional object annotations. Furthermore, leveraging the IndVisSGG method, we have implemented a meticulous annotation procedure to compile a high-quality industrial scene graph generation (ISG) dataset, comprising 10,000 images of manufacturing and related industrial scenes. Through comparisons with various scene graph generation methods and benchmarks across two other datasets, we have showcased the superiority of the IndVisSGG method and underscored the benefits of the ISG dataset over existing datasets.

关键词： Scene graph vision-language model Industrial spatial intelligence Smart manufacturing

来源：评论

学校读者我要写书评

暂无评论

Multi-MELO: Unified multimodal model editing with dynamic LoRA

引用

EXPERT SYSTEMS WITH APPLICATIONS 2025年 273卷

作者： Chen, Qin Yin, Jianghao Yu, Lang Zhou, Jie He, Liang East China Normal Univ Sch Comp Sci & Technol Shanghai 200062 Peoples R China East China Normal Univ Shanghai Inst AI Educ Shanghai 200062 Peoples R China

model editing aims to correct hallucinations or incorporate new knowledge into the pre-trained neural networks. Most previous researches focus on model editing with merely the textual modality, while editing for multimodal models is not well studied. Recent research investigates how to adapt language model editors to multimodal scenarios. However, these methods are limited to image-to-text tasks and similar model architectures. The text-to-image editing task remains unexplored, presenting significant challenges due to the diversity of complex network architectures. In this paper, we propose a unified multimodal model editing framework based on dynamic LoRA (Multi-MELO), which enables effective editing for various multimodal models by dynamically activating corresponding LoRA blocks that encode the related knowledge. We explore the framework for editing diverse multimodal models (i.e., BLIP-2, and latent diffusion model) on three downstream tasks, including image captioning, visual question answering and text-to-image generation. The experimental results show that Multi-MELO achieves superior editing performance compared to the recent state-of-the-art baselines, and meanwhile requires no extra training for additional modules.

关键词： model editing Knowledge editing vision-language model Diffusion model

来源：评论

学校读者我要写书评

暂无评论

Inference Calibration of vision-language Foundation models for Zero-Shot and Few-Shot Learning

引用

PATTERN RECOGNITION LETTERS 2025年 192卷 15-21页

作者： Hu, Minyang Chang, Hong Shan, Shiguang Chen, Xilin Chinese Acad Sci Chinese Acad Sci CAS Inst Comp Technol Key Lab Intelligent Informat Proc Beijing 100190 Peoples R China Univ Chinese Acad Sci Beijing 100049 Peoples R China

Contrastive language-Image Pre-training (CLIP) models exhibit impressive zero-shot performance across various downstream cross-modal tasks by simply computing the dot product between image and text features. CLIP is pre-trained on large-scale image-text pairs using the InfoNCE loss, which maximizes the cosine similarity of positive image-text pairs while minimizing the similarity of negative pairs. However, an objective mismatch exists between the downstream usage and the pre-training phase, as the inference phase fails to exploit information from negative samples. Intuitively, since the CLIP model has been optimized based on the InfoNCE loss, the downstream usage should also be in alignment. In this paper, we start from analyzing the InfoNCE loss and derive its upper bound. Our derivation reveals that the dot-product operation serves a zero-order approximation of this upper bound, while a centralization operation represents a first-order approximation. To address the objective mismatch problem, we propose a novel method, Inference Calibration (IC), which leverages the first-order and second-order moments of data distribution to calibrate features for zero-shot and few-shot scenarios. Experiments on various cross-modal tasks demonstrate the effectiveness of IC in both zero-shot and few-shot scenarios over dot-product operation and other comparative methods.

关键词： vision-language model Zero/Few-Shot Learning Contrastive language-Image Pre-training

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：