检索结果-内蒙古大学图书馆

Text-Guided Refinement for referring image segmentation

APPLIED SCIENCES-BASEL 2025年第9期15卷 5047-5047页

作者： Qiu, Shuang Zhang, Shiyin Ruan, Tao Taiyuan Univ Technol Sch Artificial Intelligence Taiyuan 030024 Peoples R China North China Elect Power Univ Sch Control & Comp Engn Baoding 071003 Peoples R China Beijing Jiaotong Univ State Key Lab Adv Rail Autonomous Operat Beijing 100044 Peoples R China

referring image segmentation aims to segment an object described by a natural language expression from an image. Existing methods perform multi-modal fusion during encoding, typically integrating image and text features before predicting masks via upsampling networks. However, this approach often lacks sufficient multi-modal interactions during decoding, leading to challenges in achieving precise edge predictions for objects with varying scales. Additionally, the isolated interaction between linguistic and visual features at different scales fails to utilize the continuous guidance of language to multi-scale visual features. To address this issue, we propose the Text-Guided Refinement Network (TGRN). It employs a cascaded pyramid structure with a text-guided gating mechanism to enable selective and efficient integration of multi-modal features across multiple scales at the decoding stage. The proposed TGRN offers the following advantages: (a) It enhances information flow across feature scales, improving the network's capacity to represent multi-scale semantics and achieve accurate segmentation. (b) It leverages text information to guide feature fusion, allowing for strengthened multi-modal interactions and refined edge perception during decoding. (c) It facilitates effective multi-modal information integration through a language-embedded visual encoder. Extensive experiments on three benchmark datasets validate the effectiveness of the proposed approach, demonstrating its superior performance in referring segmentation.

关键词： referring image segmentation multi-modal gating mechanism cascaded network

来源：评论

学校读者我要写书评

暂无评论

SATR: Semantics-Aware Triadic Refinement network for referring image segmentation

引用

KNOWLEDGE-BASED SYSTEMS 2024年 284卷

作者： Xie, Jialong Liu, Jin Wang, Guoxiang Zhou, Fengyu Shandong Univ Sch Control Sci & Engn Jinan Peoples R China

referring image segmentation (RIS) is a fundamental cross-modal task that aims to predict the pixel-level segmentation mask of the target referred by a natural language expression. Existing methods usually focus on leveraging word-to-pixel interaction mechanisms to directly generate final masks, while they ignore semantic alignment between the query and visual context, as well as rich fine-grained spatial details of the referred object, resulting in inaccurate identification, blurry boundary, and missing small *** address these issues, we propose a Semantics-Aware Triadic Refinement (SATR) network for referring image segmentation. Specifically, to bridge the gap between visual and linguistic modalities, we propose a Language-Guided Pixel Modulation (LGPM), which utilizes word-and sentence-level features to facilitate word -to-pixel interaction and sentence-to-object alignment, respectively. Meanwhile, the LGPM is plugged into an off-the-shelf pre-trained visual backbone to jointly learn and extract multi-modal features, which avoids the tedious learning phase of low-level features from scratch. In addition, without any post-processing for refining the final mask, we design a triadic refinement decoder to selectively extract and aggregate salient object features, pixel-level details, and boundary information, which preserves the rich spatial features to generate a high-quality mask. Further, we use a multi-task strategy to capture the target-specific context during training. Experimental results demonstrate the proposed method performs favorably against the previous approaches on the challenging RefCOCO, RefCOCO+, and RefCOCOg datasets.

关键词： referring image segmentation Triadic Refinement network Language-guided pixel modulation Multi-task learning Boundary enhancement

来源：评论

学校读者我要写书评

暂无评论

Dual-graph hierarchical interaction network for referring image segmentation

引用

DISPLAYS 2023年 80卷

作者： Shi, Zhaofeng Wu, Qingbo Li, Hongliang Meng, Fanman Ngan, King Ngi Univ Elect Sci & Technol China Sch Informat & Commun Engn Chengdu 611731 Peoples R China

referring image segmentation (RIS) aims to extract the object or stuff from an image according to the given natural language expression. As a representative multi-modal reasoning task, the main challenge of RIS lies in accurately understanding and aligning two types of heterogeneous data (i.e. image and text). Existing methods typically conduct this task via inexplicit cross-modal fusion toward the visual and linguistic features, which are separately extracted from different encoders and hard to capture accurate image-text alignment information due to their distinct latent representation structures. In this paper, we propose a Dual-Graph Hierarchical Interaction Network (DGHIN) to facilitate the explicit and comprehensive alignment between the image and text data. Firstly, two graphs are separately built for the initial visual and linguistic features extracted with different pre-trained encoders. By means of graph reasoning, we obtain a unified representation structure for different modalities to capture the intra-modal entities and their contexts, where each projected node incorporates the long-range dependencies into the latent representation. Then, the Hierarchical Interaction Module (HIM) is applied to the visual and linguistic graphs to extract comprehensive inter-modal correlations from the entity level and graph level, which not only capture the corresponding keywords and visual patches but also draws the whole sentence closer to the image region with the consistent context in the latent space. Extensive experiments on RefCOCO, RefCOCO+, G-Ref, and ReferIt demonstrate that the proposed DGHIN outperforms many state-of-the-art methods. Code is available at https://***/ZhaofengSHI/referring-DGHIN.

关键词： referring image segmentation Graph reasoning Hierarchical interaction

来源：评论

学校读者我要写书评

暂无评论

Calibration & Reconstruction: Deeply Integrated Language for referring image segmentation 24

Calibration & Reconstruction: Deeply Integrated Language for...

引用

Proceedings of the 2024 International Conference on Multimedia Retrieval

作者： Yichen Yan Xingjian He Sihan Chen Jing Liu Institute of Automation Chinese Academy of Sciences & School of Artificial Intelligence University of Chinese Academy of Sciences Beijing China Institute of Automation Chinese Academy of Sciences Beijing China School of Artificial Intelligence University of Chinese Academy of Sciences Beijing China

ISBN: (纸本)9798400706196

referring image segmentation aims to segment an object referred to by natural language expression from an image. The primary challenge lies in the efficient propagation of fine-grained semantic information from textual features to visual features. Many recent works utilize a Transformer to address this challenge. However, conventional transformer decoders can distort linguistic information with deeper layers, leading to suboptimal results. In this paper, we introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder. We start by generating language queries using vision features, emphasizing different aspects of the input language. Then, we propose a novel Calibration Decoder (CDec) wherein the multi-modal features can iteratively calibrated by the input language features. In the Calibration Decoder, we use the output of each decoder layer and the original language features to generate new queries for continuous calibration, which gradually updates the language features. Based on CDec, we introduce a Language Reconstruction Module and a reconstruction loss. This module leverages queries from the final layer of the decoder to reconstruct the input language and compute the reconstruction loss. This can further prevent the language information from being lost or distorted. Our experiments consistently show the superior performance of our approach across RefCOCO, RefCOCO+, and G-Ref datasets compared to state-of-the-art methods.

关键词： iterative calibration language reconstruction referring image segmentation

来源：评论

学校读者我要写书评

暂无评论

CMF: CASCADED MULTI-MODEL FUSION FOR referring image segmentation

CMF: CASCADED MULTI-MODEL FUSION FOR REFERRING IMAGE SEGMENT...

引用

IEEE International Conference on image Processing (ICIP)

作者： Yang, Jianhua Huang, Yan Ma, Zhanyu Wang, Liang Beijing Univ Posts & Thlecommun Sch Artificial Intelligence PRIS Lab Beijing Peoples R China Chinese Acad Sci CASIA Inst Automat Ctr Res Intelligent Percept & Comp CRIPC Natl Lab Pattern Recognit NLPR Beijing Peoples R China Beijing Acad Artificial Intelligence Beijing Peoples R China Chinese Acad Sci AIR CAS Artificial Intelligence Res Beijing Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing Peoples R China

ISBN: (纸本)9781665441155

In this work, we address the task of referring image segmentation (RIS), which aims at predicting a segmentation mask for the object described by a natural language expression. Most existing methods focus on establishing unidirectional or directional relationships between visual and linguistic features to associate two modalities together, while the multi-scale context is ignored or insufficiently modeled. Multi-scale context is crucial to localize and segment those objects that have large scale variations during the multi-modal fusion process. To solve this problem, we propose a simple yet effective Cascaded Multi-modal Fusion (CMF) module, which stacks multiple atrous convolutional layers in parallel and further introduces a cascaded branch to fuse visual and linguistic features. The cascaded branch can progressively integrate multi-scale contextual information and facilitate the alignment of two modalities during the multi-modal fusion process. Experimental results on four benchmark datasets demonstrate that our method outperforms most state-of-the-art methods. Code is available at https://***/jianhua2022/CMF-Refseg.

关键词： referring image segmentation Natural Language Expression Context Modeling

来源：评论

学校读者我要写书评

暂无评论

Two-stage Visual Cues Enhancement Network for referring image segmentation 21

Two-stage Visual Cues Enhancement Network for Referring Imag...

引用

29th ACM International Conference on Multimedia (MM)

作者： Jiao, Yang Jie, Zequn Luo, Weixin Chen, Jingjing Jiang, Yu-Gang Wei, Xiaolin Ma, Lin Fudan Univ Shanghai Key Lab Intelligent Informat Proc Sch Comp Sci Shanghai Peoples R China Meituan Beijing Peoples R China

ISBN: (纸本)9781450386517

referring image segmentation (RIS) aims at segmenting the target object from an image referred by one given natural language expression. The diverse and flexible expressions as well as complex visual contents in the images raise the RIS model with higher demands for investigating fine-grained matching behaviors between words in expressions and objects presented in images. However, such matching behaviors are hard to be learned and captured when the visual cues of referents (i.e. referred objects) are insufficient, as the referents with weak visual cues tend to be easily confused by cluttered background at boundary or even overwhelmed by salient objects in the image. And the insufficient visual cues issue can not be handled by the cross-modal fusion mechanisms as done in previous work. In this paper, we tackle this problem from a novel perspective of enhancing the visual information for the referents by devising a Two-stage Visual cues enhancement Network (TV-Net), where a novel Retrieval and Enrichment Scheme (RES) and an Adaptive Multi-resolution feature Fusion (AMF) module are proposed. Specifically, RES retrieves the most relevant image from an external data pool with regard to both the visual and textual similarities, and then enriches the visual information of the referent with the retrieved image for better multimodal feature learning. AMF further enhances the visual detailed information by incorporating the highresolution feature maps from lower convolution layers of the image. Through the two-stage enhancement, our proposed TV-Net enjoys better performances in learning fine-grained matching behaviors between the natural language expression and image, especially when the visual information of the referent is inadequate, thus produces better segmentation results. Extensive experiments are conducted to validate the effectiveness of the proposed method on the RIS task, with our proposed TV-Net surpassing the state-of-theart approaches on four benchmark datasets.

关键词： referring image segmentation retrieval and enrichment scheme adaptive multi-resolution feature fusion

来源：评论

学校读者我要写书评

暂无评论

Dual Convolutional LSTM Network for referring image segmentation

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2020年第12期22卷 3224-3235页

作者： Ye, Linwei Liu, Zhi Wang, Yang Univ Manitoba Dept Comp Sci Winnipeg MB R3T 2N2 Canada Shanghai Univ Shanghai Inst Adv Commun & Data Sci Shanghai 200444 Peoples R China Shanghai Univ Sch Commun & Informat Engn Shanghai 200444 Peoples R China

We consider referring image segmentation. It is a problem at the intersection of computer vision and natural language understanding. Given an input image and a referring expression in the form of a natural language sentence, the goal is to segment the object of interest in the image referred by the linguistic query. To this end, we propose a dual convolutional LSTM (ConvLSTM) network to tackle this problem. Our model consists of an encoder network and a decoder network, where ConvLSTM is used in both encoder and decoder networks to capture spatial and sequential information. The encoder network extracts visual and linguistic features for each word in the expression sentence, and adopts an attention mechanism to focus on words that are more informative in the multimodal interaction. The decoder network integrates the features generated by the encoder network at multiple levels as its input and produces the final precise segmentation mask. Experimental results on four challenging datasets demonstrate that the proposed network achieves superior segmentation performance compared with other state-of-the-art methods.

关键词： image segmentation Visualization Decoding Linguistics Task analysis Logic gates Computer vision referring image segmentation encoder-decoder vision and language deep learning

来源：评论

学校读者我要写书评

暂无评论

A CONTEXT-BASED NETWORK FOR referring image segmentation

A CONTEXT-BASED NETWORK FOR REFERRING IMAGE SEGMENTATION

引用

IEEE International Conference on image Processing (ICIP)

作者： Li, Xinyu Liu, Yu Xu, Kaiping Zhao, Zhehuan Liu, Sipei Dalian Univ Technol Sch Software Dalian 116620 Peoples R China North Informat Control Res Acad Grp Co Ltd Nanjing 211153 Peoples R China

ISBN: (纸本)9781728163956

referring image segmentation is an important task aiming at segmenting out the object referred by a natural language expression. Current works usually employ the methods of concatenating the visual and linguistic features. They underestimate the importance of language-to-vision and object-to-object relationships when the natural language expression has multiple entities. Therefore, we propose a new network named Context-Based Network(CBN) to improve the accuracy of locating the correct referent. The CBN is composed of two modules: Intra Relation Selection(Intra-RS) and Inter Relation Selection(Inter-RS). The Intra-RS can capture object-to-object relationships in an embedding visual and linguistic feature space and the Inter-RS uses the multi-scale linguistic features as a guide to match the most similar region from the image feature maps. Besides, we apply spatial pyramid pooling to get global information to solve the limited receptive field problem. Experimental results on four public datasets showed that CBN achieved comparable performance to the other state-of-art methods.

关键词： referring image segmentation Convolutional Neural Networks Dense Convolution

来源：评论

学校读者我要写书评

暂无评论

Exploring Fine-Grained image-Text Alignment for referring Remote Sensing image segmentation

引用

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 2025年 63卷

作者： Lei, Sen Xiao, Xinyu Zhang, Tianlin Li, Heng-Chao Shi, Zhenwei Zhu, Qing Southwest Jiaotong Univ Sch Informat Sci & Technol Chengdu 611756 Peoples R China Co Ant Grp Hangzhou 688688 Peoples R China AVIC Luoyang Inst Electroopt Equipment Luoyang 471000 Peoples R China Beihang Univ Image Proc Ctr Sch Astronaut State Key Lab Virtual Real Technol & Syst Beijing 100191 Peoples R China Southwest Jiaotong Univ Fac Geosci & Engn Chengdu 611756 Peoples R China

Given a language expression, referring remote sensing image segmentation (RRSIS) aims to identify ground objects and assign pixelwise labels within the imagery. One of the key challenges for this task is to capture discriminative multimodal features via image-text alignment. However, the existing RRSIS methods use one vanilla and coarse alignment, where the language expression is directly extracted to be fused with the visual features. In this article, we argue that a "fine-grained image-text alignment" can improve the extraction of multimodal information. To this point, we propose a new RRSIS method to fully exploit the visual and linguistic representations. Specifically, the original referring expression is regarded as context text, which is further decoupled into the ground object and spatial position texts. The proposed fine-grained image-text alignment module (FIAM) would simultaneously leverage the features of the input image and the corresponding texts, obtaining better discriminative multimodal representation. Meanwhile, to handle the various scales of ground objects in remote sensing, we introduce a text-aware multiscale enhancement module (TMEM) to adaptively perform cross-scale fusion and intersections. We evaluate the effectiveness of the proposed method on two public referring remote sensing datasets including RefSegRS and RRSIS-D, and our method obtains superior performance over several state-of-the-art methods. The code will be publicly available at https://***/Shaosifan/FIANet.

关键词： Remote sensing image segmentation Visualization Feature extraction Linguistics Transformers Electronic mail Adaptation models Object recognition Grounding Fine-grained image-text alignment referring image segmentation remote sensing images

来源：评论

学校读者我要写书评

暂无评论

Multimodal-Aware Fusion Network for referring Remote Sensing image segmentation

引用

IEEE GEOSCIENCE AND REMOTE SENSING LETTERS 2025年 22卷

作者： Shi, Leideng Zhang, Juan Shanghai Univ Engn Sci Sch Elect & Elect Engn Shanghai 201600 Peoples R China

referring remote sensing image segmentation (RRSIS) is a novel visual task in remote sensing images segmentation, which aims to segment objects based on a given text description, with great significance in practical application. Previous studies fuse visual and linguistic modalities by explicit feature interaction, which fail to effectively excavate useful multimodal information from dual-branch encoder. In this letter, we design a multimodal-aware fusion network (MAFN) to achieve fine-grained alignment and fusion between the two modalities. We propose a correlation fusion module (CFM) to enhance multiscale visual features by introducing adaptive noise in transformer, and integrate cross-modal aware features. In addition, MAFN employs multiscale refinement convolution (MSRC) to adapt to the various orientations of objects at different scales to boost their representation ability to enhances segmentation accuracy. Extensive experiments have shown that MAFN is significantly more effective than the state of the art (SOTA) on RRSIS-D datasets. The source code is available at https://***/Roaxy/MAFN.

关键词： Visualization image segmentation Remote sensing Convolution Transformers Feature extraction Noise Correlation Linguistics Accuracy Multimodal feature fusion referring image segmentation remote sensing images semantic segmentation Swin Transformer

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：