检索结果-内蒙古大学图书馆

Vision-Aware Language Reasoning for referring image segmentation

NEURAL PROCESSING LETTERS 2023年第8期55卷 11313-11331页

作者： Xu, Fayou Luo, Bing Zhang, Chao Xu, Li Pu, Mingxing Li, Bo Xihua Univ Sch Comp & Software Engn Chengdu 610039 Peoples R China Sichuan Police Coll Key Lab Intelligent Policing Luzhou 646000 Peoples R China Xihua Univ Sch Sci Chengdu 610039 Peoples R China

referring image segmentation is a multimodal joint task that aims to segment linguistically indicated objects from images in paired expressions and images. However, the diversity of language annotations trends to result in semantic ambiguity, which makes the semantic representation of language feature encoding imprecise. Existing methods ignore the correction of language encoding module, so that the semantic error of language features cannot be improved in the subsequent process, resulting in semantic deviation. To this end, we propose a vision-aware language reasoning model. Intuitively, the segmentation result can be used to guide the reconstruction of language features, which could be expressed as a tree-structured recursive process. Specifically, we designed a language reasoning encoding module and a mask loopback optimization module to optimize the language encoding tree. The feature weights of tree nodes are learned through backpropagation. In order to overcome the problem that local language words and visual regions are easily introduced into noise regions in the traditional attention module, we use the global language prior information to calculate the importance of different words to further weight the visual region features, which could be embodied as language-aware vision attention module. Our experimental results on four benchmark datasets show that the proposed method achieves performance improvement.

关键词： referring image segmentation Vision and language Explainable language-structure reasoning

来源：评论

学校读者我要写书评

暂无评论

Cross-modal transformer with language query for referring image segmentation

引用

NEUROCOMPUTING 2023年 536卷 191-205页

作者： Zhang, Wenjing Tan, Quange Li, Pengxin Zhang, Qi Wang, Rong Peoples Publ Secur Univ China Sch Informat & Cyber Secur Beijing 434020 Peoples R China Minist Publ Secur Key Lab Secur Prevent Technol & Risk Assessment Beijing 434020 Peoples R China

referring image segmentation (RIS) aims to predict a segmentation mask for a target specified by a nat-ural language expression. However, the existing methods failed to implement deep interaction between vision and language is needed in RIS, resulting inaccurate segmentation. To address the problem, a cross -modal transformer (CMT) with language queries for referring image segmentation is proposed. First, a cross-modal encoder of CMT is designed for intra-modal and inter-modal interaction, capturing context-aware visual features. Secondly, to generate compact visual-aware language queries, a language-query encoder (LQ) embeds key visual cues into linguistic features. In particular, the combina-tion of the cross-modal encoder and language query encoder realizes the mutual guidance of vision and language. Finally, the cross-modal decoder of CMT is constructed to learn multimodal features of the ref-erent from the context-aware visual features and visual-aware language queries. In addition, a semantics-guided detail enhancement (SDE) module is constructed to fuse the semantic-rich multimodal features with detail-rich low-level visual features, which supplements the spatial details of the predicted segmentation masks. Extensive experiments on four referring image segmentation datasets demonstrate the effectiveness of the proposed method.(c) 2023 Elsevier B.V. All rights reserved.

关键词： referring image segmentation Deep interaction Cross -modal transformer Semantics -guided detail enhancement

来源：评论

学校读者我要写书评

暂无评论

Local-global coordination with transformers for referring image segmentation

引用

NEUROCOMPUTING 2023年第1期522卷 39-52页

作者： Liu, Fang Kong, Yuqiu Zhang, Lihe Feng, Guang Yin, Baocai Dalian Univ Technol Sch Comp Sci & Technol Dalian 116024 Peoples R China Dalian Univ Technol Sch Innovat & Entrepreneurship Dalian 116024 Peoples R China Dalian Univ Technol Sch Informat & Commun Engn Dalian Peoples R China

referring image segmentation has sprung up benefiting from the outstanding performance of deep neural networks. However, most existing methods explore either local details or the global context of the scene without sufficiently modelling the coordination between them, leading to sub-optimal results. In this paper, we propose a transformer-based method to enforce the in-depth coordination between short -and long-range dependencies in both explicit and implicit fusion processes. Specifically, we design a Cross Modality Transformer (CMT) module with two successive blocks for explicitly integrating linguistic and visual features, which first locates the related visual region in a global view before concentrating on local patterns. Besides, a Hybrid Transformer Architecture (HTA) is utilized as a feature extractor in the encoding stage to capture global relationships and retain local cues. It can further aggregate the multi -modal features in an implicit manner. In the decoding stage, a Cross-level Information Integration module (CI2) is developed to gather information from adjacent levels by dual top-down paths, including a guided filtration path and a residual reservation path. Experimental results show that the proposed method out-performs the state-of-the-art methods on four RIS benchmarks.(c) 2022 Elsevier B.V. All rights reserved.

关键词： referring image segmentation Cross modality transformer Cross-level information integration

来源：评论

学校读者我要写书评

暂无评论

Multiscale deep feature selection fusion network for referring image segmentation

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2023年第12期83卷 36287页

作者： Dai, Xianwen Lin, Jiacheng Nai, Ke Li, Qingpeng Li, Zhiyong Hunan Univ Coll Comp Sci & Elect Engn Changsha Peoples R China Hunan Univ Sch Robot Changsha Hunan Peoples R China

referring image segmentation has attracted extensive attention in recent years. Previous methods have explored the difficult alignment between visual and textual features, but this problem has not been effectively addressed. This leads to the problem of insufficient interaction between visual features and textual features, which affects model performance. To this end, we propose a language-aware pixel feature fusion module (LPFFM) based on self-attention mechanism to ensure that the features of the two modalities have sufficient interaction in the space and channels. Then we apply it in the shallow to deep layers of the encoder to gradually select visual features related to the text. Secondly, we propose a second selection mechanism to further select visual features that only contain the target. For this mechanism, we design an attention contrastive loss to better suppress irrelevant background information. Further, we propose a multi-scale deep features selection fusion network (MDSFNet) based on the U-net architecture. Finally, the experimental results show that our proposed method is competitive with previous methods, improving the performance by 2.87%, 3.17%, and 3.81% on three benchmark datasets, RefCOCO, RefCOCO+, and G-ref, respectively.

关键词： referring image segmentation Semantic segmentation Multi-modal fusion Deep learning

来源：评论

学校读者我要写书评

暂无评论

Multi-Modal Mutual Attention and Iterative Interaction for referring image segmentation

引用

IEEE TRANSACTIONS ON image PROCESSING 2023年 32卷 3054-3065页

作者： Liu, Chang Ding, Henghui Zhang, Yulun Jiang, Xudong Nanyang Technol Univ Sch Elect & Elect Engn EEE Singapore 639798 Singapore Swiss Fed Inst Technol Comp Vis Lab CVL CH-8092 Zurich Switzerland

We address the problem of referring image segmentation that aims to generate a mask for the object specified by a natural language expression. Many recent works utilize Transformer to extract features for the target object by aggregating the attended visual regions. However, the generic attention mechanism in Transformer only uses the language input for attention weight calculation, which does not explicitly fuse language features in its output. Thus, its output feature is dominated by vision information, which limits the model to comprehensively understand the multi-modal information, and brings uncertainty for the subsequent mask decoder to extract the output mask. To address this issue, we propose Multi-Modal Mutual Attention (M(3)Att) and Multi-Modal Mutual Decoder (M(3)Dec) that better fuse information from the two input modalities. Based on M(3)Dec, we further propose Iterative Multi-modal Interaction (IMI) to allow continuous and in-depth interactions between language and vision features. Furthermore, we introduce Language Feature Reconstruction (LFR) to prevent the language information from being lost or distorted in the extracted feature. Extensive experiments show that our proposed approach significantly improves the baseline and outperforms state-of-the-art referring image segmentation methods on RefCOCO series datasets consistently.

关键词： Transformers Decoding image segmentation Task analysis Feature extraction image reconstruction Iterative methods referring image segmentation multi-modal mutual attention iterative multi-modal interaction language feature reconstruction

来源：评论

学校读者我要写书评

暂无评论

Cross-modal attention guided visual reasoning for referring image segmentation

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2023年第19期82卷 28853-28872页

作者： Zhang, Wenjing Hu, Mengnan Tan, Quange Zhou, Qianli Wang, Rong Peoples Publ Secur Univ China Sch Informat & Cyber Secur Beijing 434020 Peoples R China Shandong Police Coll Police Technol & Equipment Innovat Res Ctr Jinan 250200 Peoples R China Minist Publ Secur Key Lab Secur Prevent Technol & Risk Assessment Beijing 434020 Peoples R China

The goal of referring image segmentation (RIS) is to generate the foreground mask of the object described by a natural language expression. The key of RIS is to learn the valid multimodal features between visual and linguistic modalities to identify the referred object accurately. In this paper, a cross-modal attention-guided visual reasoning model for referring segmentation is proposed. First, the multi-scale detailed information is captured by a pyramidal convolution module to enhance visual representation. Then, the entity words of the referring expression and relevant image regions are aligned by a cross-modal attention mechanism. Based on this, all the entities described by the expression can be identified. Finally, a fully connected multimodal graph is constructed with multimodal features and relationship cues of expressions. Visual reasoning is performed stepwisely on the graph to highlight the correct entity whiling suppressing other irrelevant ones. The experiment results on four benchmark datasets show that the proposed method achieves performance improvement (e.g., +1.13% on UNC, +3.06% on UNC+, +2.1% on G-Ref, and 1.11% on ReferIt). Also, the effectiveness and feasibility of each component of our method are verified by extensive ablation studies.

关键词： referring image segmentation Multi-scales features Cross-modal attention mechanism Graph convolution

来源：评论

学校读者我要写书评

暂无评论

Text-Guided image Manipulation via Generative Adversarial Network With referring image segmentation-Based Guidance

引用

IEEE ACCESS 2023年 11卷 42534-42545页

作者： Watanabe, Yuto Togo, Ren Maeda, Keisuke Ogawa, Takahiro Haseyama, Miki Hokkaido Univ Grad Sch Informat Sci & Technol Sapporo 0600814 Japan Hokkaido Univ Fac Informat Sci & Technol Sapporo 0600814 Japan

This study proposes a novel text-guided image manipulation method that introduces referring image segmentation into a generative adversarial network. The proposed text-guided image manipulation method aims to manipulate images containing multiple objects while preserving text-unrelated regions. The proposed method assigns the task of distinguishing between text-related and unrelated regions in an image to segmentation guidance based on referring image segmentation. With this architecture, the adversarial generative network can focus on generating new attributes according to the text description and reconstructing text-unrelated regions. For the challenging input images with multiple objects, the experimental results demonstrate that the proposed method outperforms conventional methods in terms of image manipulation precision.

关键词： image segmentation Text recognition Generative adversarial networks image color analysis Visualization image reconstruction Text processing Text-guided image manipulation text-to-image synthesis generative adversarial network referring image segmentation

来源：评论

学校读者我要写书评

暂无评论

Dual Context Perception Transformer for referring image segmentation 7th

Dual Context Perception Transformer for Referring Image Segm...

引用

7th Chinese Conference on Pattern Recognition and Computer Vision

作者： Kong, Yuqiu Liu, Junhua Yao, Cuili Dalian Univ Technol Dalian 116024 Peoples R China

ISBN: (纸本)9789819786190;9789819786206

referring image segmentation segments target objects in the image according to language expressions. Existing methods mainly make efforts to integrate multi-modal features with attention mechanisms. However, most methods tend to incline to the feature of a single modal during the fusion stage and fall short in exploring cross-modal contextual information, which is critical in localizing accurate target regions. To this end, we propose a novel architecture named Dual Context Perception Transformer (DCPformer) which considers both visual and linguistic contextual information during the fusion and reasoning stages. Specifically, a Cross-modal Context-aware Perception Module (CCPM) is designed to model cross-modal alignment in a unified visual-linguistic representation space. Furthermore, we propose an Information Feedback Module (IFM) that generates a rectification mask based on deep-scale features and filters unrelated signals of the target object in features of shallower scales. Extensive experiments show that the proposed DCP-former achieves state-of-the-art performances against existing methods on three challenging benchmarks.

关键词： referring image segmentation Vision-linguistic alignment Multi-modal fusion

来源：评论

学校读者我要写书评

暂无评论

SMVT: Spectrum-Driven Multi-scale Vision Transformer for referring image segmentation 20th

SMVT: Spectrum-Driven Multi-scale Vision Transformer for Ref...

引用

20th International Conference on Intelligent Computing (ICIC)

作者： Li, Tianxiao Chen, Junhong Huang, Yiheng Huang, Kesi Xia, Qiqiang Asim, Muhammad Liu, Wenyin Guangdong Univ Technol Fac Comp Sci & Technol Guangzhou 510006 Peoples R China Hasselt Univ Flanders Make Expertise Ctr Digital Media Hasselt Belgium Zhongguancun Lab Beijing Peoples R China

ISBN: (纸本)9789819755967;9789819755974

referring image segmentation is a challenging task at the intersection of computer vision and natural language processing, aiming to segment out an object referred to by a natural language expression from an image. Recently despite significant progress on this task, existing methods still face challenges in effectively integrating visual and language information and enhancing the model's ability to capture fine-grained details within images. These challenges primarily originate from a lack of a mechanism capable of deeply and comprehensively fusing visual features with language features and effectively utilizing cross-modal features. To address these problems, we propose the Spectrum-driven Multi-scale Visual Transformer (SMVT), which incorporates two innovative designs: Spectrum-driven Fusion Attention (SFA) and the Cross-modal Feature Refinement Enhancement (CFRE) module. SFA, by guiding the fusion of visual and linguistic features at the spectral domain level, effectively captures fine-grained features in images and enhances the model's sensitivity to local spectral domain information, thereby responding more accurately to the detail requirements in language descriptions. CFRE module, by refining and enhancing cross-modal features at different layers, enhances the complementarity and the ability to capture fine-grained cross-modal features across different layers, promoting the precise alignment of visual and language features. These two modules enable the SMVT to more effectively process visual and language information. Experiments on three benchmark datasets have shown that our method surpasses state-of-the-art approaches.

关键词： referring image segmentation Cross-modal learning Spectrum-driven fusion

来源：评论

学校读者我要写书评

暂无评论

Fuse and Calibrate: A Bi-directional Vision-Language Guided Framework for referring image segmentation 20th

Fuse and Calibrate: A Bi-directional Vision-Language Guided ...

引用

20th International Conference on Intelligent Computing (ICIC)

作者： Yan, Yichen He, Xingjian Chen, Sihan Lu, Shichen Liu, Jing Chinese Acad Sci Inst Automat Beijing Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing Peoples R China Beihang Univ Sch Comp Sci & Engn Beijing Peoples R China

ISBN: (纸本)9789819756117;9789819756124

referring image segmentation (RIS) aims to segment an object described in natural language from an image, with the main challenge being a text-to-pixel correlation. Previous methods typically rely on single-modality features, such as vision or language features, to guide the multi-modal fusion process. However, this approach limits the interaction between vision and language, leading to a lack of fine-grained correlation between the language description and pixel-level details during the decoding process. In this paper, we introduce FCNet, a framework that employs a bi-directional guided fusion approach where both vision and language play guiding roles. Specifically, we use a vision-guided approach to conduct initial multi-modal fusion, obtaining multi-modal features that focus on key vision information. We then propose a language-guided calibration module to further calibrate these multi-modal features, ensuring they understand the context of the input sentence. This bi-directional vision-language guided approach produces higher-quality multi-modal features sent to the decoder, facilitating adaptive propagation of fine-grained semantic information from textual features to visual features. Experiments on RefCOCO, RefCOCO+, and G-Ref datasets with various backbones consistently show our approach outperforming state-of-the-art methods.

关键词： referring image segmentation Vision-Language Models Fusion & Calibration

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：