检索结果-内蒙古大学图书馆

ReferSAM: Unleashing Segment Anything Model for referring image segmentation

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2025年第5期35卷 4910-4922页

作者： Liu, Sun-Ao Xie, Hongtao Ge, Jiannan Zhang, Yongdong Univ Sci & Technol China Sch Informat Sci & Technol Hefei 230022 Peoples R China

The Segment Anything Model (SAM) has demonstrated remarkable capability as a general segmentation model given visual prompts such as points or boxes. While SAM is conceptually compatible with text prompts, it merely employs linguistic features from vision-language models as prompt embeddings and lacks fine-grained cross-modal interaction. This deficiency limits its application in referring image segmentation (RIS), where the targets are specified by free-form natural language expressions. In this paper, we introduce ReferSAM, a novel SAM-based framework that enhances cross-modal interaction and reformulates prompt encoding, thereby unleashing SAM's segmentation capability for RIS. Specifically, ReferSAM incorporates the Vision-Language Interactor (VLI) to integrate linguistic features with visual features during the image encoding stage of SAM. This interactor introduces fine-grained alignment between linguistic features and multi-scale visual representations without altering the architecture of pre-trained models. Additionally, we present the Vision-Language Prompter (VLP) to generate dense and sparse prompt embeddings by aggregating the aligned linguistic and visual features. Consequently, the generated embeddings sufficiently prompt SAM's mask decoder to provide precise segmentation results. Extensive experiments on five public benchmarks demonstrate that ReferSAM achieves state-of-the-art performance on both classic and generalized RIS tasks. The code and models are available at https://***/lsa1997/ReferSAM.

关键词： Visualization image segmentation Decoding Linguistics Feature extraction Computational modeling Vectors Transformers image coding Circuits and systems referring image segmentation segment anything vision-language interactor prompt learning

来源：评论

学校读者我要写书评

暂无评论

Mixed-scale cross-modal fusion network for referring image segmentation

引用

NEUROCOMPUTING 2025年 614卷

作者： Pan, Xiong Xie, Xuemei Yang, Jianxiu Xidian Univ Sch Artificial Intelligence Xian 710071 Peoples R China Xidian Univ Guangzhou Inst Technol Guangzhou 510700 Peoples R China Pazhou Lab Huangpu Guangzhou 510555 Peoples R China Shanxi Datong Univ Datong 037009 Peoples R China

referring image segmentation aims to segment the target by a given language expression. Recently, the bottom-up fusion network utilizes language features to highlight the most relevant regions during the visual encoder stage. However, it is not comprehensive that establish only the relationship between pixels and words. To alleviate this problem, we propose a mixed-scale cross-modal fusion method that widens the interaction between vision and language. Specially, at each stage, pyramid pooling is used to augment visual perception and improve the interaction between visual and linguistic features, thereby highlighting relevant regions in the visual data. Additionally, we employ a simple multi-scale feature fusion module to effectively combine multi-scale aligned features. Experiments conducted on Standard RIS benchmarks demonstrate that the proposed method achieves favorable performance against state-of-the-art approaches. Moreover, we conducted experiments on different visual backbones respectively, and the proposed method yielded better and significantly improved performance results.

关键词： referring image segmentation Mixed-scale cross-modal fusion Cross-modality transformer

来源：评论

学校读者我要写书评

暂无评论

Prompt-guided bidirectional deep fusion network for referring image segmentation

引用

NEUROCOMPUTING 2025年 616卷

作者： Wu, Junxian Zhang, Yujia Kampffmeyer, Michael Zhao, Xiaoguang Chinese Acad Sci Inst Automat State Key Lab Multimodal Artificial Intelligence S Beijing Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing Peoples R China UiT Arctic Univ Norway Dept Phys & Technol Tromso Norway

referring image segmentation involves accurately segmenting objects based on natural language descriptions. This poses challenges due to the intricate and varied nature of language expressions, as well as the requirement to identify relevant image regions among multiple objects. Current models predominantly employ language- aware early fusion techniques, which may lead to misinterpretations of language expressions due to the lack of explicit visual guidance of the language encoder. Additionally, early fusion methods are unable to adequately leverage high-level contexts. To address these limitations, this paper introduces the Prompt-guided Bidirectional Deep Fusion Network (PBDF-Net) to enhance the fusion of language and vision modalities. In contrast to traditional unidirectional early fusion approaches, our approach employs a prompt-guided bidirectional encoder fusion (PBEF) module to promote mutual cross-modal fusion across multiple stages of the vision and language encoders. Furthermore, PBDF-Net incorporates a prompt-guided cross-modal interaction (PCI) module during the late fusion stage, facilitating amore profound integration of contextual information from both modalities, resulting in more accurate target segmentation. Comprehensive experiments conducted on the RefCOCO, RefCOCO+, G-Ref and ReferIt datasets substantiate the efficacy of our proposed method, demonstrating significant advancements in performance compared to existing approaches.

关键词： referring image segmentation Prompt-guided bidirectional encoder fusion Prompt-guided cross-modal interaction

来源：评论

学校读者我要写书评

暂无评论

Cross-Aware Early Fusion With Stage-Divided Vision and Language Transformer Encoders for referring image segmentation

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 5823-5833页

作者： Cho, Yubin Yu, Hyunwoo Kang, Suk-Ju Sogang Univ Sch Artificial Intelligence Seoul 04017 South Korea Sogang Univ Sch Elect Engn Seoul 04017 South Korea

referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.

关键词： referring image segmentation cross-aware early fusion feature-based cross-modal alignment

来源：评论

学校读者我要写书评

暂无评论

CM-MaskSD: Cross-Modality Masked Self-Distillation for referring image segmentation

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 6906-6916页

作者： Wang, Wenxuan He, Xingjian Zhang, Yisi Guo, Longteng Shen, Jiachen Li, Jiangyun Liu, Jing Chinese Acad Sci Inst Automat Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100190 Peoples R China Univ Sci & Technol Beijing Sch Automat & Elect Engn Beijing 100083 Peoples R China

referring image segmentation (RIS) is a fundamental vision-language task that intends to segment a desired object from an image based on a given natural language expression. Due to the essentially distinct data properties between image and text, most of existing methods either introduce complex designs towards fine-grained vision-language alignment or lack required dense alignment, resulting in scalability issues or mis-segmentation problems such as over- or under-segmentation. To achieve effective and efficient fine-grained feature alignment in the RIS task, we explore the potential of masked multimodal modeling coupled with self-distillation and propose a novel cross-modality masked self-distillation framework named CM-MaskSD, in which our method inherits the transferred knowledge of image-text semantic alignment from CLIP model to realize fine-grained patch-word feature alignment for better segmentation accuracy. Moreover, our CM-MaskSD framework can considerably boost model performance in a nearly parameter-free manner, since it shares weights between the main segmentation branch and the introduced masked self-distillation branches, and solely introduces negligible parameters for coordinating the multimodal features. Comprehensive experiments on three benchmark datasets (i.e. RefCOCO, RefCOCO+, G-Ref) for the RIS task convincingly demonstrate the superiority of our proposed framework over previous state-of-the-art methods.

关键词： referring image segmentation cross-modality guidance masked self-distillation vision and language

来源：评论

学校读者我要写书评

暂无评论

Area-keywords cross-modal alignment for referring image segmentation

引用

NEUROCOMPUTING 2024年 581卷

作者： Zhang, Huiyong Wang, Lichun Li, Shuang Xu, Kai Yin, Baocai Beijing Univ Technol Beijing Key Lab Multimedia & Intelligent Software Beijing 100124 Peoples R China Beijing Univ Technol Beijing Artificial Intelligence Inst Fac Informat Technol Beijing 100124 Peoples R China Beijing Informat Sci & Technol Univ Sch Automat Beijing 100192 Peoples R China Beijing Univ Technol 100 Pingleyuan Beijing 100124 Peoples R China

referring image segmentation aims to segment the instance corresponding to the given language description, which requires aligning information from two modalities. Existing approaches usually align the cross -modal information based on different forms of feature units, such as pixel -sentence, pixel -word and patch -word. The problem is that the semantic information embodied by these feature units may be mismatched, for example, the semantics transferred by a pixel is a part of the semantics of a sentence. When using this inconsistent information to model the relationship between feature units from two modalities, the obtained relationship between the modes are imprecise, resulting in inaccurate cross -modal features. In this paper, we propose to generate scalable area and keywords features to ensure that the feature units from the two modalities have comparable semantic granularity. Meanwhile, the scalable features provide sparse representation for image and text, which reduces computation complexity for computing cross -modal features. In addition, we design a novel multi -source driven dynamic convolution to inversely map the area -keywords cross -modal features to image to predicate mask. The experimental results on three benchmark datasets demonstrate that our proposed framework achieves advanced performance, and the calculation amount of the model has been greatly reduced.

关键词： referring image segmentation Cross-modal alignment Dynamic convolution

来源：评论

学校读者我要写书评

暂无评论

Text-Guided Refinement for referring image segmentation

引用

APPLIED SCIENCES-BASEL 2025年第9期15卷 5047-5047页

作者： Qiu, Shuang Zhang, Shiyin Ruan, Tao Taiyuan Univ Technol Sch Artificial Intelligence Taiyuan 030024 Peoples R China North China Elect Power Univ Sch Control & Comp Engn Baoding 071003 Peoples R China Beijing Jiaotong Univ State Key Lab Adv Rail Autonomous Operat Beijing 100044 Peoples R China

referring image segmentation aims to segment an object described by a natural language expression from an image. Existing methods perform multi-modal fusion during encoding, typically integrating image and text features before predicting masks via upsampling networks. However, this approach often lacks sufficient multi-modal interactions during decoding, leading to challenges in achieving precise edge predictions for objects with varying scales. Additionally, the isolated interaction between linguistic and visual features at different scales fails to utilize the continuous guidance of language to multi-scale visual features. To address this issue, we propose the Text-Guided Refinement Network (TGRN). It employs a cascaded pyramid structure with a text-guided gating mechanism to enable selective and efficient integration of multi-modal features across multiple scales at the decoding stage. The proposed TGRN offers the following advantages: (a) It enhances information flow across feature scales, improving the network's capacity to represent multi-scale semantics and achieve accurate segmentation. (b) It leverages text information to guide feature fusion, allowing for strengthened multi-modal interactions and refined edge perception during decoding. (c) It facilitates effective multi-modal information integration through a language-embedded visual encoder. Extensive experiments on three benchmark datasets validate the effectiveness of the proposed approach, demonstrating its superior performance in referring segmentation.

关键词： referring image segmentation multi-modal gating mechanism cascaded network

来源：评论

学校读者我要写书评

暂无评论

Bilateral Knowledge Interaction Network for referring image segmentation

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 2966-2977页

作者： Ding, Haixin Zhang, Shengchuan Wu, Qiong Yu, Songlin Hu, Jie Cao, Liujuan Ji, Rongrong Xiamen Univ Key Lab Multimedia Trusted Percept & Efficient Com Minist Educ China Xiamen 361005 Peoples R China

referring image segmentation aims to segment objects that are described by natural language expressions. Although remarkable advancements have been made to align natural language expressions with visual representations for better performance, the interaction between image-level and text-level information is still not formulated properly. Most of the previous works focus on building correlations between vision and language, ignoring the variety of objects. The target objects with unique appearances may not be correctly located or completely segmented. In this article, we propose a novel Bilateral Knowledge Interaction Network, termed BKINet, which reformulates the image-text interaction in a bilateral manner to adapt concrete knowledge of the target object in the image. BKINet contains two key components: a knowledge learning module (KLM) and a knowledge applying module (KAM). In the KLM, the abstract knowledge from text features is replenished with concrete knowledge from visual features to adapt to the target objects in the input images, which generates the knowledge interaction kernels (KI kernels) containing abundant referring information. With the referring information of KI kernels, the KAM is designed to highlight the most relevant visual features for predicting the accurate segmentation mask. Extensive experiments on three widely-used datasets, i.e. RefCOCO, RefCOCO+, and G-ref, demonstrate the superiority of BKINet over the state-of-the-art.

关键词： image segmentation Visualization Kernel Knowledge engineering Feature extraction Semantics Convolution referring image segmentation vision-language

来源：评论

学校读者我要写书评

暂无评论

A survey of methods for addressing the challenges of referring image segmentation

引用

NEUROCOMPUTING 2024年 583卷

作者： Ji, Lixia Du, Yunlong Dang, Yiping Gao, Wenzhao Zhang, Han Zhengzhou Univ Sch Cyber Sci & Engn Zhengzhou 450002 Henan Peoples R China Sichuan Univ Coll Comp Sci Chengdu 610041 Sichuan Peoples R China Sichuan Police Coll Intelligent Policing Key Lab Sichuan Prov Luzhou 646000 Sichuan Peoples R China Ulster Univ Intelligent Syst Res Ctr Sch Comp Engn & Intelligent Syst Magee Campus Londonderry BT487JL North Ireland

referring image segmentation is guided by natural language descriptions to separate the target objects in an image. This task is different from semantic segmentation and instance segmentation in that it involves unique challenges such as multimodal information fusion, variability of natural language expressions, and model robustness. In recent years, the emergence of deep learning techniques has led to innovative ideas and methods for solving these problems. We systematically analyze the main challenges of referring image segmentation and summarize the existing solutions. These include strategies such as multimodal fusion, expression query, multimodal pre -training, and robustness. In addition, we provide an overview of several datasets commonly used in referring image segmentation and analyze the performance of various representative approaches in comparison to different datasets, visual backbone models and threshold settings. Our focus also extends to the challenges and future developments in the field of referring image segmentation. Our survey paper will provide a comprehensive technical reference for future researchers.

关键词： referring image segmentation Multi-modal learning Computer vision Natural language processing

来源：评论

学校读者我要写书评

暂无评论

Cross-Modal Recurrent Semantic Comprehension for referring image segmentation

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2023年第7期33卷 3229-3242页

作者： Shang, Chao Li, Hongliang Qiu, Heqian Wu, Qingbo Meng, Fanman Zhao, Taijin Ngan, King Ngi Univ Elect Sci & Technol China Sch Informat & Commun Engn Chengdu 611731 Peoples R China

referring image segmentation aims to segment the target object from the image according to the description of language expression. Due to the diversity of language expressions, word sequences in different orders often express different semantic information. The previous methods focus more on matching different words to different visual regions in the image separately, ignoring the global semantic understanding of language expression based on the sequence structure. To address this problem, we redesign a new recurrent network structure for referring image segmentation, called Cross-Modal Recurrent Semantic Comprehension Network (CRSCNet), to obtain a more comprehensive global semantic understanding through iterative cross-modal semantic reasoning. Specifically, in each iteration, we first propose a Dynamic SepConv to extract relevant visual features guided by language and further propose Language Attentional Feature Modulation to improve the feature discriminability, then propose a Cross-Modal Semantic Reasoning module to perform global semantic reasoning by capturing both linguistic and visual information, and finally updates and corrects the visual features of the predicted object based on semantic information. Moreover, we further propose a Cross-Modal ASPP to capture richer visual information referred to in the global semantics of the language expression from larger receptive fields. Extensive experiments demonstrate that our proposed network significantly outperforms previous state-of-the-art methods on multiple datasets.

关键词： referring image segmentation cross-modal recurrent network global semantic reasoning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：