检索结果-内蒙古大学图书馆

Mamba-driven hierarchical temporal multimodal alignment for referring video object segmentation

NEUROCOMPUTING 2025年 622卷

作者： Liang, Le Zhang, Lefei Wuhan Univ Sch Comp Sci Wuhan 430072 Peoples R China

The objective of referring video object segmentation (RVOS) is to segment target objects within video sequences described by natural language expressions. Previous approaches often overlook the latent temporal information in the language modality, and they fail to align this information with the temporal dimension of the visual modality, leading to a lack of comprehensive cross-modal understanding between visual and linguistic spaces. Additionally, the absence of pixel-level inter-frame information aggregation hinders finegrained space-time consistency modeling of the target object, thereby reducing segmentation accuracy. To address these issues in RVOS, we propose a novel mamba-driven hierarchical temporal multimodal alignment framework, abbreviated as MHTMA. Specifically, we introduce a hierarchical multimodal interaction and temporal integration module designed to enhance the temporal correlation of multimodal features, achieving cross-modal temporal alignment of visual and linguistic features to enable more accurate decoding of the target object. Moreover, we develop a mamba-based temporal representation learning module that learns the spatio-temporal features of the referred object at the pixel level, resulting in more precise generation of spatiotemporally consistent target object masks. Extensive experimental results on four benchmarks demonstrate that our MHTMA outperforms state-of-the-art methods, validating the effectiveness of the proposed modules.

关键词： referring video object segmentation State space model Multimodal alignment Temporal modeling

来源：评论

学校读者我要写书评

暂无评论

Weakly Supervised referring video object segmentation With object-Centric Pseudo-Guidance

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2025年 27卷 1320-1333页

作者： Wang, Weikang Su, Yuting Liu, Jing Sun, Wei Zhai, Guangtao Tianjin Univ Sch Elect & Informat Engn Tianjin 300072 Peoples R China Shanghai Jiao Tong Univ Inst Image Commun & Network Engn Shanghai 200240 Peoples R China Shanghai Jiao Tong Univ AI Inst Inst Image Commun & Network Engn MoE Key Lab Artificial Intelligence Shanghai 200240 Peoples R China

referring video object segmentation (RVOS) is an emerging task for multimodal video comprehension while the expensive annotating process of object masks restricts the scalability and diversity of RVOS datasets. To relax the dependency on expensive mask annotations and take advantage from large-scale partially annotated data, in this paper, we explore a novel extended RVOS task, namely weakly supervised referring video object segmentation (WRVOS), which employs multiple weak supervision sources, including object points and bounding boxes. Correspondingly, we propose a unified WRVOS framework. Specifically, an object-centric pseudo mask generation method is introduced to provide effective shape priors for the pseudo guidance of spatial object location. Then, a pseudo-guided optimization strategy is proposed to effectively optimize the object outlines in terms of spatial location and projection density with a multi-stage online learning strategy. Furthermore, a multimodal cross-frame level set evolution method is proposed to iteratively refine the object boundaries considering both temporal consistency and cross-modal interactions. Extensive experiments are conducted on four publicly available RVOS datasets, including A2D Sentences, J-HMDB Sentences, Ref-DAVIS, and Ref-YoutubeVOS. Performance comparison shows that the proposed method achieves state-of-the-art performance in both point-supervised and box-supervised settings.

关键词： Shape object segmentation Weak supervision Semantics Level set Transformers Annotations Feature extraction Supervised learning Optimization referring video object segmentation weakly supervised segmentation pseudo-guided learning level set evolution

来源：评论

学校读者我要写书评

暂无评论

Expression Prompt Collaboration Transformer for universal referring video object segmentation

引用

KNOWLEDGE-BASED SYSTEMS 2025年 311卷

作者： Chen, Jiajun Lin, Jiacheng Zhong, Guojin Fu, Haolong Nai, Ke Yang, Kailun Li, Zhiyong Hunan Univ Sch Robot Changsha 410082 Peoples R China Hunan Univ Coll Comp Sci & Elect Engn Changsha 410082 Peoples R China Changsha Univ Sci & Technol Sch Comp & Commun Engn Changsha 410011 Peoples R China

Audio-guided video object segmentation (A-VOS) and referring video object segmentation (R-VOS) are two highly related tasks aiming to segment specific objects from video sequences according to expression prompts. However, due to the challenges of modeling representations for different modalities, existing methods struggle to balance between interaction flexibility and localization precision. In this paper, we address this problem from two perspectives: the alignment of audio and text and the deep interaction among audio, text, and visual modalities. First, we propose a universal architecture, the Expression Prompt Collaboration Transformer, herein EPCFormer. Next, we propose an Expression Alignment (EA) mechanism for audio and text. The proposed EPCFormer exploits the fact that audio and text prompts referring to the same objects are semantically equivalent by using contrastive learning for both expressions. Then, to facilitate deep interactions among audio, text, and visual modalities, we introduce an Expression-Visual Attention (EVA) module. The knowledge of video object segmentation in terms of the expression prompts can seamlessly transfer between the two tasks by deeply exploring complementary cues between text and audio. Experiments on well-recognized benchmarks demonstrate that our EPCFormer attains state-of-the-art results on both tasks.

关键词： Audio-guided video object segmentation referring video object segmentation Expression-visual attention Audio-text contrastive learning Multi-task learning

来源：评论

学校读者我要写书评

暂无评论

Multi-Context Temporal Consistent Modeling for referring video object segmentation

Multi-Context Temporal Consistent Modeling for Referring Vid...

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Choi, Sun-Hyuk Jo, Hayoung Lee, Seong-Whan Department of Artificial Intelligence Korea University Seoul Korea Republic of

ISBN: (纸本)9798350368741

referring video object segmentation aims to segment objects within a video corresponding to a given text description. Existing transformer-based temporal modeling approaches face challenges related to query inconsistency and the limited consideration of context. Query inconsistency produces unstable masks of different objects in the middle of the video. The limited consideration of context leads to the segmentation of incorrect objects by failing to adequately account for the relationship between the given text and instances. To address these issues, we propose the Multi-context Temporal Consistency Module (MTCM), which consists of an Aligner and a Multi-Context Enhancer (MCE). The Aligner removes noise from queries and aligns them to achieve query consistency. The MCE predicts text-relevant queries by considering multi-context. We applied MTCM to four different models, increasing performance across all of them, particularly achieving 47.6 J&F on the MeViS. Code is available at https://***/Choi58/MTCM. © 2025 IEEE.

关键词： multi-context referring video object segmentation temporal consistency

来源：评论

学校读者我要写书评

暂无评论

Decoupling Multimodal Transformers for referring video object segmentation

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR video TECHNOLOGY 2023年第9期33卷 4518-4528页

作者： Gao, Mingqi Yang, Jinyu Han, Jungong Lu, Ke Zheng, Feng Montana, Giovanni Southern Univ Sci & Technol Dept Comp Sci & Engn Shenzhen 518055 Peoples R China Univ Warwick Warwick Mfg Grp Coventry CV1 7AL England Univ Birmingham Sch Comp Sci Birmingham B15 2TT England Univ Sheffield Dept Comp Sci Sheffield S10 2TN England Univ Chinese Acad Sci Sch Engn Sci Beijing 100049 Peoples R China Peng Cheng Lab Shenzhen 518055 Peoples R China

referring video object segmentation (RVOS) aims to segment the text-depicted object from video sequences. With excellent capabilities in long-range modelling and information interaction, transformers have been increasingly applied in existing RVOS architectures. To better leverage multimodal data, most efforts focus on the interaction between visual and textual features. However, they ignore the syntactic structures of the text during the interaction, where all textual components are intertwined, resulting in ambiguous vision-language alignment. In this paper, we improve the multimodal interaction by DECOUPLING the interweave. Specifically, we train a lightweight subject perceptron, which extracts the subject part from the input text. Then, the subject and text features are fed into two parallel branches to interact with visual features. This enables us to perform subject-aware and context aware interactions, respectively, thus encouraging more explicit and discriminative feature embedding and alignment. Moreover, we find the decoupled architecture also facilitates incorporating the vision-language pre-trained alignment into RVOS, further improving the segmentation performance. Experimental results on all RVOS benchmark datasets demonstrate the superiority of our proposed method over the state-of-the-arts. The code of our method is available at: https://***/gaomingqi/dmformer.

关键词： Decoupled multimodal transformers referring video object segmentation vision-language pre-training

来源：评论

学校读者我要写书评

暂无评论

Temporally Consistent referring video object segmentation With Hybrid Memory

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR video TECHNOLOGY 2024年第11期34卷 11373-11385页

作者： Miao, Bo Bennamoun, Mohammed Gao, Yongsheng Shah, Mubarak Mian, Ajmal Univ Western Australia Dept Comp Sci & Software Engn Perth WA 6009 Australia Griffith Univ Sch Engn Brisbane Qld 4111 Australia Univ Cent Florida Ctr Res Comp Vision Orlando FL 32816 USA

referring video object segmentation (R-VOS) methods face challenges in maintaining consistent object segmentation due to temporal context variability and the presence of other visually similar objects. We propose an end-to-end R-VOS paradigm that explicitly models temporal instance consistency alongside the referring segmentation. Specifically, we introduce a novel hybrid memory that facilitates inter-frame collaboration for robust spatio-temporal matching and propagation. Features of frames with automatically generated high-quality reference masks are propagated to segment the remaining frames based on multi-granularity association to achieve temporally consistent R-VOS. Furthermore, we propose a new Mask Consistency Score (MCS) metric to evaluate the temporal consistency of video segmentation. Extensive experiments demonstrate that our approach enhances temporal consistency by a significant margin, leading to top-ranked performance on popular R-VOS benchmarks, i.e., Ref-YouTube-VOS (67.1%) and Ref-DAVIS17 (65.6%). The code is available at https://***/bo-miao/HTR.

关键词： referring video object segmentation temporal consistency deep learning

来源：评论

学校读者我要写书评

暂无评论

LEVERAGING VISUAL PROMPTS TO GUIDE LANGUAGE MODELING FOR referring video object segmentation 30

LEVERAGING VISUAL PROMPTS TO GUIDE LANGUAGE MODELING FOR REF...

引用

30th IEEE International Conference on Image Processing (ICIP)

作者： Gao, Qiqi Zhong, Wanjun Li, Jie Zhao, Tiejun Harbin Inst Technol Harbin Peoples R China Sun Yat Sen Univ Guangzhou Peoples R China

ISBN: (纸本)9781728198354

referring video object segmentation (R-VOS) aims to segment object masks in a target video given a language query describing the object. It is a challenging task that requires modeling the semantics of a natural language query and its correspondence to the target video. Previous works directly use visual-agnostic language features from uni-modal language models, and only interact with visual features in late decoding stages. We propose to encode visual-enriched language features by using visual prompts as guidance in the early encoding stage. The proposed visual prompt is constructed by modulating visual features of key frames with alignment scores to text inputs. The alignment score is computed with a pre-trained visual-language contrastive model. We concatenate visual prompts with text inputs to encode visual-enriched language features, which serve as queries for target object segmentation in a Transformer-based decoder. Our method outperforms the previous state-of-the-art method (+2.3) on Refer-Youtube-VOS benchmark.

关键词： multi-modal referring video object segmentation prompt

来源：评论

学校读者我要写书评

暂无评论

Leveraging Hybrid referring Expressions for referring video object segmentation

Leveraging Hybrid Referring Expressions for Referring Video ...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Li, Yan Wang, Qiong Zhejiang Univ Technol Coll Comp Sci & Technol Hangzhou Peoples R China Shenzhen Univ Guangdong Prov Key Lab Intelligent Informat Proc Shenzhen Peoples R China Shenzhen Univ Shenzhen Key Lab Media Secur Shenzhen Peoples R China

ISBN: (纸本)9798350390155;9798350390162

Existing referring video object segmentation methods segment the target object referred by one referring expression or a simple group of multiple referring expressions by word concatenation in all video frames, which might still suffer from the issue of wrong object identification. To solve this problem, we propose a new transformer-based framework built upon hybrid referring expressions. To fully explore the multi-model fusion, three interaction methods are studied, and the early interaction shows the best alignment between discriminative video features and multi-language semantics. Moreover, two ways are explored to generate conditional queries in transformer-based methods, and the proposed word-group level method can better represent the target instance. Experiments on the large-scale Ref-YoutubeVOS benchmark show that the proposed model achieves 63.8% J & F, exceeding the state-of-the-art methods.

关键词： referring video object segmentation Multimodel fusion object query Transformer

来源：评论

学校读者我要写书评

暂无评论

MRRVOS: Modular Refinement referring video object segmentation 5th

MRRVOS: Modular Refinement Referring Video Object Segmentati...

引用

5th Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) International Joint Conference on Web and Big Data

作者： Duan, Zhijiang Sun, Yukuan Wang, Jianming TianGong Univ Sch Comp Sci & Technol Tianjin 300387 Peoples R China TianGong Univ Ctr Engn Intership & Training Tianjin 300387 Peoples R China

ISBN: (纸本)9789811681431;9789811681424

The previous referring video object segmentation method only focuses on the final prediction result. When the user input linguistic query does not match the information in the video, the video still segments error information. We propose a new reference video object segmentation framework. In our model, which we call the Modular Refinement referring video object segmentation (MRRVOS), when the objects in the referring linguistic query do not match the video frame, stop segmentation and feedback error information. Firstly, given a video clip and a linguistic query. Our method segments the specified object in a video frame automatically. Then our method matches linguistic query with video frames and implements object segmentation on other frames through a recursive three-module model: (1) referring video object segmentation module: we consider the referring video object segmentation task as a joint problem of referring object segmentation in the image and mask propagation in the video. (2) Image caption module: using recurrent neural networks (RNNs), and deep convolutional neural network (CNN) to encode every frame except the first frame, and a Long Short Term Memory (LSTM) RNN decoder to generate the output caption and put it into the corpus. (3) Semantic dissimilarity module: put all the text results into the corpus and embed vector space, our linguistic query of the input to perform a semantic dissimilarity search. We show that our approach is competitive to the state-of-the-art method.

关键词： referring video object segmentation Semantic similarly Image caption

来源：评论

学校读者我要写书评

暂无评论

Fine-grained Text-video Fusion for referring video object segmentation 4

Fine-grained Text-Video Fusion for Referring Video Object Se...

引用

4th IEEE International Conference on Power, Electronics and Computer Applications, ICPECA 2024

作者： Li, Yonglin Wan, Jiaqi Teng, Xiao Lan, Long College of Computer Science and Technology National University of Defense Technology Department of Intelligent Data Science Changsha410073 China

ISBN: (纸本)9798350359589

referring video object segmentation (RVOS) aims at segmenting an object in a video with its text description. The core of RVOS lies in the modal alignment between the vision and text. To improve the performance, most previous RVOS methods are devoted to exploring more sophisticated visual cues. However, these methods fail to fully exert the inherent structured properties of text and only use coarse text features to tackle vision-text interaction. In this paper, we propose FTVR, an approach that utilizes Fine-grain Text-video method for referring video object segmentation to explore fine-grain text information. We introduce a LLM to split the text sentence into different functional phrases and propose three novel modules to enhance the cross-modal alignment. Concretely, we design a dynamic-aware perception module to deal with motion-related phrases and a following global-aware attention module to fuse the outputted motion information. To deal with entity-related phrases, our FTVR also introduces an entity-aware augmentation module to highlight entity information. Extensive experiments show the effectiveness of our method on four popular benchmarks. © 2024 IEEE.

关键词： fine grained fusion large language model referring video object segmentation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：