检索结果-内蒙古大学图书馆

referring image segmentation With Fine-Grained Semantic Funneling Infusion

IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024年第10期35卷 14727-14738页

作者： Yang, Jiaxing Zhang, Lihe Lu, Huchuan Dalian Univ Technol Sch Informat & Commun Engn Dalian 116024 Peoples R China

Recently, referring image segmentation has attracted wide attention given its huge potential in human-robot interaction. Networks to identify the referred region must have a deep understanding of both the image and language semantics. To do so, existing works tend to design various mechanisms to achieve cross-modality fusion, for example, tile and concatenation and vanilla nonlocal manipulation. However, the plain fusion usually is either coarse or constrained by the exorbitant computation overhead, finally causing not enough understanding of the referent. In this work, we propose a fine-grained semantic funneling infusion (FSFI) mechanism to solve the problem. The FSFI introduces a constant spatial constraint on the querying entities from different encoding stages and dynamically infuses the gleaned language semantic into the vision branch. Moreover, it decomposes the features from different modalities into more delicate components, allowing the fusion to happen in multiple low-dimensional spaces. The fusion is more effective than the one only happening in one high-dimensional space, given its ability to sink more representative information along the channel dimension. Another problem haunting the task is that the instilling of high-abstract semantic will blur the details of the referent. Targetedly, we propose a multiscale attention-enhanced decoder (MAED) to alleviate the problem. We design a detail enhancement operator (DeEh) and apply it in a multiscale and progressive way. Features from the higher level are used to generate attention guidance to enlighten the lower-level features to more attend to the detail regions. Extensive results on the challenging benchmarks show that our network performs favorably against the state-of-the-arts (SOTAs).

关键词： Index Terms- Detail enhancement operator (DeEh) fine-grained semantic funneling infusion (FSFI) multiscale attention enhanced decoder (MAED) referring image segmentation

来源：评论

学校读者我要写书评

暂无评论

referring image segmentation with Multi-Modal Feature Interaction and Alignment Based on Convolutional Nonlinear Spiking Neural Membrane Systems

引用

INTERNATIONAL JOURNAL OF NEURAL SYSTEMS 2024年第12期34卷 2450064页

作者： Sun, Siyan Wang, Peng Peng, Hong Liu, Zhicai Xihua Univ Sch Comp & Software Engn 999 Jinzhou Rd Chengdu Sichuan Peoples R China

referring image segmentation aims to accurately align image pixels and text features for object segmentation based on natural language descriptions. This paper proposes NSNPRIS (convolutional nonlinear spiking neural P systems for referring image segmentation), a novel model based on convolutional nonlinear spiking neural P systems. NSNPRIS features NSNPFusion and Language Gate modules to enhance feature interaction during encoding, along with an NSNPDecoder for feature alignment and decoding. Experimental results on RefCOCO, RefCOCO+, and G-Ref datasets demonstrate that NSNPRIS performs better than mainstream methods. Our contributions include advances in the alignment of pixel and textual features and the improvement of segmentation accuracy.

关键词： Convolutional nonlinear spiking neural P systems referring image segmentation feature alignment feature interaction

来源：评论

学校读者我要写书评

暂无评论

referring image segmentation FOR REMOTE SENSING DATA

REFERRING IMAGE SEGMENTATION FOR REMOTE SENSING DATA

引用

IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

作者： Yuan, Zhenghang Mou, Lichao Hua, Yuansheng Zhu, Xiao Xiang Tech Univ Munich TUM Data Sci Earth Observat Munich Germany Shenzhen Univ Coll Civil & Transportat Engn Shenzhen Peoples R China

ISBN: (纸本)9798350360332;9798350360325

In this paper, we present a new task: referring image segmentation for remote sensing data, which targets segmenting out specific objects referred to by natural language. Due to the absence of a dataset for this task, we construct a dataset based on the SkyScapes dataset. Our dataset is designed with linguistically structured expressions that focus on object categories, attributes, and spatial relationships, enabling the generation of binary masks from semantic segmentation maps. To benchmark this task, we evaluate and compare the performance of three different convolutional neural network (CNN)-based methods and a Transformer-based method. Experimental results provide valuable insights into the adaptability of these methods to remote sensing data, highlighting the potential of our dataset as a resource for the remote sensing community to further explore vision-language tasks.

关键词： referring image segmentation remote sensing vision-language task

来源：评论

学校读者我要写书评

暂无评论

referring image segmentation Without Text Annotations 20th

Referring Image Segmentation Without Text Annotations

引用

20th International Conference on Intelligent Computing (ICIC)

作者： Liu, Jing Jiang, Huajie Bi, Yandong Hu, Yongli Yin, Baocai Beijing Univ Technol Beijing Inst Artificial Intelligence Fac Informat Technol Beijing Key Lab Multimedia & Intelligent Software Beijing 100124 Peoples R China

ISBN: (纸本)9789819756148;9789819756155

referring image segmentation (RIS) is an essential topic in visual language understanding that aims to segment the target instance in the image referred to by the language description. Conventional RIS methods have relied on expensive manual annotations involving the triplet (image-text-mask), with the acquisition of text annotations posing the most formidable challenge. To eliminate the heavy dependence on human annotations, we propose a novel RIS method, the referring image segmentation without Text Annotations (WoTA), which substitutes textual annotations by generating the pseudo-query through the utilization of visual information. Specifically, we design a novel training-testing scheme that introduces a Pseudo-Query Generation Scheme (PQGS) in the training phase, which relies on the pre-trained cross-modal knowledge in CLIP to generate the pseudo-query related to global and local visual information. In the testing phase, the CLIP text encoder is directly applied to the test statements to generate real query language features. Extensive experiments on several benchmark datasets demonstrate the advantage of the proposed WoTA over several zero-shot baselines of the task and even the weakly supervised referring image segmentation method.

关键词： referring image segmentation Without Text Annotation Pseudo-Query

来源：评论

学校读者我要写书评

暂无评论

referring image segmentation WITH TWO-STAGE MULTI-MODAL INTERACTION 31

REFERRING IMAGE SEGMENTATION WITH TWO-STAGE MULTI-MODAL INTE...

引用

2024 International Conference on image Processing

作者： Wang, Zhenhua Ye, Linwei Wenzhou Univ Coll Comp Sci & Artificial Intelligence Wenzhou Peoples R China

ISBN: (纸本)9798350349405;9798350349399

The objective of referring image segmentation is to extract referred entities from an image using a particular natural language sentence. The main idea for this task is interacting textual and visual features to build multi-modal relationships. The prior state-of-the-art methods mainly focus on local multi-level intermediate feature interaction or global text-to-image alignment, which might result in insufficient interaction for capturing global multi-modal information exchange or fine-grained referred object details, respectively. To overcome this issue, we introduce a referring image segmentation framework with two-stage multi-modal interaction. Specifically, we devise an innovative multi-level cross-modal fusion module to effectively facilitate the interaction of intermediate features of linguistic and visual modalities for fine-grained details of referred objects. Besides, we further align the linguistic and visual information by introducing an elaborate global alignment module for accurately localizing the entire referred objects. The comprehensive experiments conducted on three referring image segmentation datasets illustrate that our proposed two-stage multi-modal interaction framework exhibits a marked superiority over the contemporary state-of-the-art approaches.

关键词： Vision and Language referring image segmentation

来源：评论

学校读者我要写书评

暂无评论

PRNet: A Progressive Refinement Network for referring image segmentation

引用

NEUROCOMPUTING 2025年 630卷

作者： Liu, Jing Jiang, Huajie Hu, Yongli Yin, Baocai Beijing Univ Technol Beijing Inst Artificial Intelligence Fac Informat Technol Beijing Key Lab Multimedia & Intelligent Software 100 Pingleyuan Beijing 100124 Peoples R China

The effective feature alignment between language and image is necessary for correctly inferring the location of reference instances in the referring image segmentation (RIS) task. Previous studies usually resort to assisting target localization with the help of external detectors or using a coarse-grained positional prior during multimodal feature fusion to implicitly enhance the modal alignment capability. However, these approaches are either limited by the performance of the external detector and the design of the matching algorithm, or ignore the fine-grained features in the reference information when using the coarse-grained prior processing, which may lead to inaccurate segmentation results. In this paper, we propose anew RIS network, Progressive Refinement Network (PRNet), which aims to gradually improve the alignment quality between language and image from coarse to fine. The core of the PRNet is the Progressive Refinement Localization Scheme (PRLS), which consists of a Coarse Positional Prior Module (CPPM) and a Refined Localization Module (RLM). The CPPM obtains rough prior positional information and corresponding semantic features by calculating the similarity matrix between sentence and image. The RLM fuses information from the visual and language modalities by densely aligning pixels with word features and utilizes the prior positional information generated by the CPPM to enhance the textual semantic understanding, thus guiding the model to perceive the position of the reference instance more accurately. Experimental results show that the proposed PRNet performs well on all three public datasets, RefCOCO, RefCOCO+, and RefCOCOg.

关键词： referring image segmentation Position prior Features alignment Progressive localization Transformer

来源：评论

学校读者我要写书评

暂无评论

Mask prior generation with language queries guided networks for referring image segmentation

引用

COMPUTER VISION AND image UNDERSTANDING 2025年 253卷

作者： Zhou, Jinhao Xiao, Guoqiang Lew, Michael S. Wu, Song Southwest Univ Coll Comp & Informat Sci Chongqing 400715 Peoples R China Southwest Univ Yibin Acad Yibin 644000 Sichuan Peoples R China Leiden Univ LIACS Media Lab Leiden Netherlands

The aim of referring image segmentation (RIS) is to generate a pixel-level mask to accurately segment the target object according to its natural language expression. Previous RIS methods ignore exploring the significant language information in both the encoder and decoder stages, and simply use an upsampling-convolution operation to obtain the prediction mask, resulting in inaccurate visual object locating. Thus, this paper proposes a Mask Prior Generation with Language Queries Guided Network (MPG-LQGNet). In the encoder of MPGLQGNet, a Bidirectional Spatial Alignment Module (BSAM) is designed to realize the bidirectional fusion for both vision and language embeddings, generating additional language queries to understand both the locating of targets and the semantics of the language. Moreover, a Channel Attention Fusion Gate (CAFG) is designed to enhance the exploration of the significance of the cross-modal embeddings. In the decoder of the MPG-LQGNet, the Language Query Guided Mask Prior Generator (LQPG) is designed to utilize the generated language queries to activate significant information in the upsampled decoding features, obtaining the more accurate mask prior that guides the final prediction. Extensive experiments on RefCOCO series datasets show that our method consistently improves over state-of-the-art methods. The source code of our MPG-LQGNet is available at https://***/SWU-CS-MediaLab/MPG-LQGNet.

关键词： referring image segmentation Bidirectional spatial alignment Channel attention fusion gate Mask prior generator

来源：评论

学校读者我要写书评

暂无评论

Hierarchical collaboration for referring image segmentation

引用

NEUROCOMPUTING 2025年 613卷

作者： Zhang, Wei Cheng, Zesen Chen, Jie Gao, Wen Harbin Inst Technol Sch Comp Sci & Technol Shenzhen 518055 Peoples R China Peng Cheng Lab Shenzhen 518000 Peoples R China Peking Univ Sch Elect & Comp Engn Shenzhen 518055 Peoples R China

In the field of referring segmentation, top-down methods and bottom-up methods are the two prevailing approaches. Both of these methods inevitably exhibit certain drawbacks. Top-down methods are susceptible to Polar Negative (PN) errors due to their limited understanding of multi-modal fine-grained features. Bottom-up methods lack macro-level object positional information, making them susceptible to Inferior Positive (IP) errors. However, we find that the two approaches are highly complementary in addressing their respective weaknesses, but combining them directly through a simple average does not yield complementary advantages. Therefore, we proposed a hierarchical collaboration approach to explore the complementary characteristics of the existing two methods from the perspectives of fusion and interaction, aiming to achieve more precise segmentation results. We proposed the Complementary Feature Interaction (CFI) module, which enables top-down methods to access fine-grained information and allows bottom-up approaches to obtain object positional information interactively. Regarding integration, Gaussian Scoring Integration (GSI) models the Gaussian performance distributions of two branches and performs weighted integration by sampling confidence scores from these distributions. We integrate various top-down and bottom-up methods within the proposed architecture and conduct experiments on three standard datasets. The experimental results demonstrate that our method outperforms the state-of-theart independent segmentation algorithms. On the RefCOCO validation, test A and test B datasets, our proposed method achieved IoU scores of 77.51, 79.12, and 72.79, respectively. Extensive experiments demonstrate that our method can significantly improve segmentation accuracy when fusing different sub-methods.

关键词： referring image segmentation image understanding Cross-modal

来源：评论

学校读者我要写书评

暂无评论

CMIRNet: Cross-Modal Interactive Reasoning Network for referring image segmentation

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2025年第4期35卷 3234-3249页

作者： Xu, Mingzhu Xiao, Tianxiang Liu, Yutong Tang, Haoyu Hu, Yupeng Nie, Liqiang Shandong Univ Sch Software Jinan 250101 Shandong Peoples R China Harbin Inst Technol Shenzhen Sch Comp Sci & Technol Shenzhen 518055 Guangdong Peoples R China

referring image segmentation (RIS) aims to semantically segment the target object (referent) in alignment with the provided natural language query. Existing works still suffer from that the non-referent was segmented mistakenly, which can be attributed to the insufficient comprehension of vision and language. To tackle this problem, we propose a Cross-Modal Interactive Reasoning Network (CMIRNet) to explore semantic information that consistently existed between vision and language. Specifically, we first devise a novel Text-Guided Multi-Modality Joint Encoder (TGMM-JE), where the key expression can be extracted and the important visual features will be encoded under the continuous guidance of language expression. Then, we design a Cross-Graph Interactive Positioning (CGIP) module to locate the key pixels of the referent object in deepest layer. The multi-modality graph data is constructed between visual and linguistic features, and the important pixels can be positioned from cross-graph interaction and intra-graph reasoning. Finally, a novel Cross-Modal Attention Enhanced DEcoder (CMAE-DE) is dedicated to refine the referent object mask from coarse to fine progressively, where hybrid cross modal attentions are explored to enhance the representation of referent object. Extensive ablation studies validate the efficacy of our key modules and comprehensive experimental results show the superiority of our proposed model over 22 state-of-the-art (SOTA) models.

关键词： Visualization Feature extraction Semantics Linguistics Cognition Decoding Circuits and systems Encoding Semantic segmentation Computer architecture referring image segmentation vision and language cross modal reasoning graph neural network

来源：评论

学校读者我要写书评

暂无评论

De-noising mask transformer for referring image segmentation

引用

image AND VISION COMPUTING 2025年 154卷

作者： Wang, Yehui Lei, Fang Wang, Baoyan Zhang, Qiang Zhen, Xiantong Zhang, Lei Guangdong Univ Petrochem Technol Maoming 525000 Peoples R China South China Univ Technol Guangzhou 510006 Peoples R China Guangdong Prov Key Lab Petrochem Equipment Fault D Maoming 525000 Peoples R China

referring image segmentation (RIS) is a challenging computer vision task that involves identifying and segmenting specific objects in an image based on a natural language description. Unlike conventional segmentation methodologies, RIS needs to bridge the gap between visual and linguistic modalities to exert the semantic information provided by natural language. Most existing RIS approaches are confronted with the common issue that the intermediate predicted target region also participates in the later feature generation and parameter updating. Then the wrong prediction, especially occurs in the early training stage, will bring the gradient misleading and ultimately affect the training stability. To tackle this issue, we propose de-noising mask (DNM) transformer to fuse the cross-modal integration, a novel framework to replace the cross-attention by DNM-attention in traditional transformer. Furthermore, two kinds of DNM-attention, named mask-DNM and cluster-DNM, are proposed, where noisy ground truth information is adopted to guide the attention mechanism to produce accurate object queries, i.e., de-nosing query. Thus, DNM-attention leverages noisy ground truth information to guide the attention mechanism to produce additional de-nosing queries, which effectively avoids the gradient misleading. Experimental results show that the DNM transformer improves the performance of RIS and outperforms most existing RIS approaches on three benchmarks.

关键词： referring image segmentation De-noising mask Multi-modal fusion

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：