检索结果-内蒙古大学图书馆

Toward the unification of generative and discriminative visual foundation model: a survey

visual COMPUTER 2025年第5期41卷 3371-3412页

作者： Liu, Xu Zhou, Tong Wang, Chong Wang, Yuping Wang, Yuanxin Cao, Qinjingwen Du, Weizhi Yang, Yonghuan He, Junjun Qiao, Yu Shen, Yiqing Univ Calif Los Angeles Dept Phys & Astron Los Angeles CA USA Rice Univ Dept Comp Sci Houston TX USA Xinxiang Med Univ Coll Med Engn Xinxiang Henan Peoples R China Univ Michigan Dept Elect & Comp Engn Ann Arbor MI USA Carnegie Mellon Univ Sch Comp Sci Pittsburgh PA USA Univ Illinois Dept Comp Sci Champaign IL USA Walmart Global Tech San Bruno CA USA Santa Clara Univ Dept Comp Sci & Engn Santa Clara CA USA Shanghai AI Lab Shanghai Peoples R China Johns Hopkins Univ Dept Comp Sci Baltimore MD 21218 USA

The advent of foundation models, which are pre-trained on vast datasets, has ushered in a new era of computer vision, characterized by their robustness and remarkable zero-shot generalization capabilities. Mirroring the transformative impact of foundation models like large language models in natural language processing, visual foundation models (VFMs) have become a catalyst for groundbreaking developments in computer vision. This review paper delineates the pivotal trajectories of VFMs, emphasizing their scalability and proficiency in generative tasks such as text-to-image synthesis, as well as their adeptness in discriminative tasks including image segmentation. While generative and discriminative models have historically charted distinct paths, we undertake a comprehensive examination of the recent strides made by VFMs in both domains, elucidating their origins, seminal breakthroughs, and pivotal methodologies. Additionally, we collate and discuss the extensive resources that facilitate the development of VFMs and address the challenges that pave the way for future research endeavors. A crucial direction for forthcoming innovation is the amalgamation of generative and discriminative paradigms. The nascent application of generative models within discriminative contexts signifies the early stages of this confluence. This survey aspires to be a contemporary compendium for scholars and practitioners alike, charting the course of VFMs and illuminating their multifaceted landscape.

关键词： Deep learning foundation model visual foundation model

来源：评论

学校读者我要写书评

暂无评论

ClassWise-SAM-Adapter: Parameter-Efficient Fine-Tuning Adapts Segment Anything to SAR Domain for Semantic Segmentation

引用

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING 2025年 18卷 4791-4804页

作者： Pu, Xinyang Jia, Hecheng Zheng, Linghao Wang, Feng Xu, Feng Fudan Univ Sch Informat Sci & Technol Key Lab Informat Sci Electromagnet Waves Minist Educ Shanghai 200433 Peoples R China

In the realm of artificial intelligence, the emergence of foundation models, backed by high computing capabilities and extensive data, has been revolutionary. A segment anything model (SAM), built on the vision transformer (ViT) model with millions of parameters and trained on its corresponding large-scale dataset SA-1B, excels in various segmentation scenarios relying on its significance of semantic information and generalization ability. Such achievement of visual foundation model stimulates continuous researches on specific downstream tasks in computer vision. The classwise-SAM-adapter (CWSAM) is designed to adapt the high-performing SAM for landcover classification on space-borne synthetic aperture radar (SAR) images. The proposed CWSAM freezes most of SAM's parameters and incorporates lightweight adapters for parameter-efficient fine-tuning, and a classwise mask decoder is designed to achieve semantic segmentation task. This adapt-tuning method allows for efficient landcover classification of SAR images, balancing the accuracy with computational demand. In addition, the task-specific input module injects low-frequency information of SAR images by MLP-based layers to improve the model performance. Compared to conventional state-of-the-art semantic segmentation algorithms by extensive experiments, CWSAM showcases enhanced performance with fewer computing resources, highlighting the potential of leveraging foundational models such as SAM for specific downstream tasks in the SAR domain.

关键词： Radar polarimetry foundation models Semantic segmentation Synthetic aperture radar Computer architecture Computational modeling Decoding Transformers Remote sensing Computer vision Adapter tuning landcover classification parameter-efficient fine-tuning segment anything (SA) synthetic aperture radar (SAR) visual foundation model

来源：评论

学校读者我要写书评

暂无评论

DefectSAM: Prototype Prompt Guided SAM for Few-Shot Defect Segmentation

引用

IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT 2025年 74卷

作者： Wu, Zhifei Zhao, Shuying Zhang, Yunzhou Jin, Yuxin Northeastern Univ Coll Informat Sci & Engn Shenyang 110819 Peoples R China

Industrial defect segmentation plays a crucial role in ensuring quality and safety within manufacturing processes. Previous research in this field has focused on training models with limited available data to enable the segmentation of new classes. However, the limited and singular training data lead to limited feature extraction capabilities of the models, hindering their generalization to other industrial scenarios. Recently, the segment anything model (SAM), an interactive visual segmentation foundation model, demonstrates remarkable zero-shot generalization capabilities. However, its performance in the specific domain of industrial defect detection is relatively poor. Therefore, we propose an approach that leverages visual foundation model to achieve few-shot defect segmentation. This method extracts prior knowledge from example images, derives multitype robust prompts from the prior knowledge, and fine-tunes SAM with few-shot learning paradigm and low-rank adaptation (LoRA) strategy. Specifically, given a few support image-mask pairs, foreground and background prototypes are extracted, and their similarities with the query image are computed to obtain prototype prior probability maps. Subsequently, a prototype prompt generator specifically for defect data extracts salient information from these prior probability maps, producing robust prototype prompts. These prototype prompts are then used to fine-tune the LoRA layers in SAM's image encoder and the lightweight mask decoder of SAM, enabling SAM to adapt to the few-shot defect segmentation task. Experimental results demonstrate that our proposed method outperforms state-of-the-art few-shot defect segmentation techniques and can be generalized to various industrial scenarios, exhibiting significant application value.

关键词： Few-shot defect segmentation low-rank adaptation (LoRA) prototype prompt segment anything visual foundation model

来源：评论

学校读者我要写书评

暂无评论

Adapting the segment anything model for multi-modal retinal anomaly detection and localization

引用

INFORMATION FUSION 2025年 113卷

作者： Li, Jingtao Chen, Ting Wang, Xinyu Zhong, Yanfei Xiao, Xuan Wuhan Univ Renmin Hosp State Key Lab Informat Engn Surveying Mapping & Re Wuhan Peoples R China Wuhan Univ Renmin Hosp Dept Ophthalmol Wuhan Peoples R China Wuhan Univ Sch Remote Sensing & Informat Engn Wuhan Peoples R China Wuhan Univ Cross Res Inst Med Remote Sensing Wuhan Peoples R China

The fusion of optical coherence tomography (OCT) and fundus modality information can provide a comprehensive diagnosis for retinal artery occlusion (RAO) disease, where OCT provides the cross-sectional examination of the fundus image. Given multi-modal retinal images, an anomaly diagnosis model can discriminate RAO without the need for real diseased samples. Despite this, previous studies have only focused on single-modal diagnosis, because of: 1) the lack of paired modality samples;and 2) the significant imaging differences, which make the fusion difficult with small-scale medical data. In this paper, we describe how we first built a multi-modal RAO dataset including both OCT and fundus modalities, which supports both the anomaly detection and localization tasks with pixel-level annotation. Motivated by the powerful generalization ability of the recent visual foundation model known as the Segment Anything model (SAM), we adapted it for our task considering the small-scale property of retinal samples. Specifically, a modality-shared decoder with task-specific tokens is introduced to make SAM support the multi-modal image setting, which includes a mask token for the anomaly localization task at the pixel level and a fusion token for the anomaly detection task at the case level. Since SAM has little medical knowledge and lacks the learning of the "normal" concept, it is infeasible to localize RAO anomalies in the zero-shot manner. To integrate expert retinal knowledge while keeping the general segmentation knowledge, general anomaly simulation for both modalities and a low-level prompt-tuning strategy are introduced. The experiments conducted in this study show that the adapted model can surpass the state-of-the-art model by a large margin. This study sets the first benchmark for the multi-modal anomaly detection and localization tasks in the medical community. The code is available at https://***/Jingtao-Li -CVer/MMRAD.

关键词： Anomaly detection Anomaly localization Retinal artery occlusion visual foundation model Multi-modal medical diagnosis Segment anything model

来源：评论

学校读者我要写书评

暂无评论

Text2Seg: Zero-shot Remote Sensing Image Semantic Segmentation via Text-Guided visual foundation models 7

Text2Seg: Zero-shot Remote Sensing Image Semantic Segmentati...

引用

7th International Workshop on AI for Geographic Knowledge Discovery

作者： Zhang, Jielu Zhou, Zhongliang Mai, Gengchen Hu, Mengxuan Guan, Zihan Li, Sheng Mu, Lan Univ Georgia Athens GA 30602 USA Univ Texas Austin Austin TX 78712 USA Univ Virginia Charlottesville VA USA

ISBN: (纸本)9798400711763

Remote sensing imagery is vital for environmental and land use monitoring, necessitating efficient segmentation methods. While deep learning models have become standard for segmentation, they require extensive per-pixel annotations, which are often scarce. High intra-dataset variance also limits the transferability of these models. Although models like SAM offer zero-shot instance-level segmentation, adapting them for semantic segmentation remains challenging. We introduce Text2Seg, a method for remote sensing semantic segmentation that reduces dependency on extensive annotations by automatically generating prompts using visual foundation models (VFMs). This approach enhances generalization across datasets. Experiments on four remote sensing datasets show Text2Seg significantly outperforms the vanilla SAM model, with improvements ranging from 31% to 225%. Our code is available at https://***/Douglas2Code/Text2Seg.

关键词： visual foundation model Remote Sensing Semantic segmentation

来源：评论

学校读者我要写书评

暂无评论

PiSSA-MoDCN: A Mixture-of-DCN for Segment Anything to Remote Sensing Ship for Semantic Segmentation 8

PiSSA-MoDCN: A Mixture-of-DCN for Segment Anything to Remote...

引用

8th International Conference on Smart Internet of Things

作者： Xia, Zhiyong Cai, Wentian Gao, Ying Chen, Yandan Ao, Yang Hou, Zhengzhi Qiu, Qin South China Univ Technol Sch Comp Sci & Engn Guangzhou 510006 Peoples R China Antidrug Tethnol Ctr Guangdong Prov Guangzhou 510030 Peoples R China

ISBN: (纸本)9798350366457;9798350366440

The Segment Anything model (SAM), with its vast array of parameters and training data, possesses a rich semantic information and excellent generalization capabilities, performing remarkably across a variety of scenarios. This has inspired exploration into its potential applications in downstream tasks. In this paper, we propose PiSSA-MoDCN, a simple yet effective parameter-efficient fine-tuning method that aligns SAM with the field of remote sensing ship image processing. By employing Singular Value Decomposition (SVD) to initialize LoRA, it retains SAM's segmentation capabilities without introducing new parameters, thereby accelerating training speed. The designed MoDCN module leverages both Mixture of Experts (MoE) and deformable convolutional networks to enhance local image priors, infusing information from remote sensing ship images to improve model performance. Through extensive comprehensive experiments, compared to traditional advanced semantic segmentation algorithms and PEFT methods, PiSSA-MoDCN achieves higher performance with minimal computational resources, highlighting the potential of utilizing foundational models like SAM for specific downstream tasks in the field of remote sensing ship image processing.

关键词： Remote Sensing Ship Segment Anything visual foundation model parameter efficient fine-tuning Deformable Convolutional Networks Mixture of Experts

来源：评论

学校读者我要写书评

暂无评论

AM-RADIO: Agglomerative Vision foundation model Reduce All Domains Into One

AM-RADIO: Agglomerative Vision Foundation Model Reduce All D...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Ranzinger, Mike Heinrich, Greg Kautz, Jan Molchanov, Pavlo NVIDIA Santa Clara CA 95051 USA

ISBN: (纸本)9798350353006

A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks. VFMs like CLIP, DINOv2, SAM are trained with distinct objectives, exhibiting unique characteristics for various downstream tasks. We find that despite their conceptual differences, these models can be effectively merged into a unified model through multi-teacher distillation. We name this approach AM-RADIO (Agglomerative model - Reduce All Domains Into One). This integrative approach not only surpasses the performance of individual teacher models but also amalgamates their distinctive features, such as zero-shot vision-language comprehension, detailed pixel-level understanding, and open vocabulary segmentation capabilities. Additionally, in pursuit of the most hardware-efficient backbone, we evaluated numerous architectures in our multi-teacher distillation pipeline using the same training recipe. This led to the development of a novel architecture (E-RADIO) that exceeds the performance of its predecessors and is at least 6x faster than the teacher models at matched resolution. Our comprehensive benchmarking process covers downstream tasks including ImageNet classification, semantic segmentation linear probing, COCO object detection and integration into LLaVa-1.5. Code: https://***/NVlabs/RADIO.

关键词： Distillation Knowledge Distillation Semantic Segmentation visual foundation model visual Large Language models Zero Shot Classification

来源：评论

学校读者我要写书评

暂无评论

Instance Segmentation for Active Sonar Image based on Image Segmentation foundation model

Instance Segmentation for Active Sonar Image based on Image ...

引用

OCEANS Conference

作者： Liu, Hongji Cheng, Gaofeng Liu, Jian Univ Chinese Acad Sci Chinese Acad Sci Inst Acoust Key Lab Speech Acoust & Content Understanding Beijing 100190 Peoples R China

ISBN: (纸本)9798350362077

Through instance segmentation techniques, foreground objects in active sonar images are segmented based on their categories, aiding in the recognition and understanding of the seafloor conditions. However, due to the scarcity of datasets for active sonar image segmentation, the results of sonar image instance segmentation are not satisfactory. Recently, Segment Anything model (SAM) by Meta demonstrated remarkable segmentation capabilities leveraging its vast training dataset, Segment Anything 1-Billion mask dataset (SA-1B). Yet, SAM struggles to achieve complete segmentation of sonar image targets, lacks classification capabilities, and its results heavily depend on the type, position, and number of prompts. This paper integrates SAM with traditional instance segmentation models, using the SAM model to enhance the performance of active sonar image object detection and instance segmentation, achieving notable results.

关键词： instance segmentation active sonar image visual foundation model Segment Anything model prompt learning

来源：评论

学校读者我要写书评

暂无评论

Structered Deep visual models For Robot Manipulation 2

Structered Deep Visual Models For Robot Manipulation

引用

2nd IEEE International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics, ICIITCEE 2024

作者： Danu, Chirag Chauhan, Rahul Bhatt, Chandradeep Devliyal, Swati Gupta, Shefali Graphic Era Hill University Computer Science and Engineering Uttarakhand Dehradun India Graphic Era Deemed to Be University Computer Science and Engineering Uttarakhand Dehradun India Chitkara University Institute of Engineering and Tech Chitkara University Punjab India

ISBN: (纸本)9798350306415

In order to enhance robot manipulation tasks and motion planning, a unique layered architecture that includes visual foundation models is presented in this research. Five levels make up the framework: learning, perception, cognition, planning, and execution. We improve the robot's perception of its surroundings through the use of visual foundation models, allowing for more precise motion planning and effective task comprehension. This method enables ongoing learning and real-time modifications, which greatly enhances job execution. The outcomes of the experiments show how well the suggested framework works in a range of robot manipulation tasks and motion planning situations, underscoring its potential for real-world implementation in dynamic contexts. © 2024 IEEE.

关键词： Cognition Framework Motion Planning Ongoing Learning Real-World Implementation Robot Perception visual foundation model

来源：评论

学校读者我要写书评

暂无评论

A robust defect detection method with a generalization enhancer and cross-modality aggregator for cylinder bores

引用

ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE 2024年 136卷

作者： He, Xujie Jin, Jing Chen, Duo Zhou, Cangtian Harbin Inst Technol Sch Astronaut Harbin 150001 Peoples R China

High-quality cylinder bores in automobile engines enable drivers to respond quickly to emergencies. Automated detection methods are gradually being adopted across various industries. However, uncontrollable factors and improper preservation methods lead to various types of defects on cylinder bores, thereby causing existing highperformance detectors to exhibit not only undesirable generalizability for unseen defect types but also a certain degree of missed detection for defects of seen types, thereby allowing defective cylinder bores to flow into the market. To address these issues, we propose a foundation-model-based robust defect detection method with high generalizability for cylinder bores (RHG-Detector). Specifically, to address unseen defect categories, we propose a generalization enhancer comprising a box filter, a region extractor and a defect discriminator (DeDi) based on a foundation model to extend defect detection from a closed set to an open set. To reduce missed detections, we adopt a cross-modality aggregator to aggregate the detection results from different modalities. Additionally, we collected and annotated challenging defect classification and detection datasets for cylinder bores, named HITEngDC (Harbin Institute of Technology Engine defect classification dataset) and HIT-EngDD2 (Engine defect detection dataset-version 2), which cover nearly all types of cylinder bore defects. Extensive experiments on HITEngDC and HIT-EngDD2 demonstrate the state-of-the-art performance of RHG-Detector, with a classification accuracy of 92.0 and a mAP@50 (mean average precision under intersection over union = 0.5) score of 45.2, where the latter is increased by 6 and 3 compared to the corresponding FasterRCNN (faster region-based convolutional neural networks) and YOLOv7 (you only look once) scores, respectively.

关键词： Open-set defect detector visual foundation model Generalization enhancer Cross-modality aggregator Supervised learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：