Decoding audio related region with temporal and structural enhancement for audio visual segmentation

作     者：Geng, Qingwei Gu, Xiaodong 

作者机构：Fudan Univ Dept Elect Engn Shanghai 200438 Peoples R China 

出 版 物：《DIGITAL SIGNAL PROCESSING》 (Digital Signal Process Rev J)

年 卷 期：2025年第165卷

核心收录：

学科分类：0808[工学-电气工程] 0809[工学-电子科学与技术（可授工学、理学学位）] 08[工学] 

基　　金：National Natural Science Foundation of China 

主　　题：Audio visual learning Video segmentation Multi-modal learning Audio-visual segmentation 

摘      要：Audio-Visual Segmentation (AVS) is a task that aims to predict pixel-level masks for sound-producing objects in videos. Recent advanced AVS methods primarily focus on cross-modal interaction while often neglecting the significance of temporal modeling and precise structural prediction. To address these challenges, we propose a novel AVS framework incorporating several innovations. Firstly, we propose a Temporal Enhancement Module (TEM) that effectively captures temporal relationships across frames. Secondly, we devise an Audio-Visual Decoder that utilizes audio information to selectively emphasize relevant visual regions during decoding. Besides, Structural Similarity (SSIM) is introduced into the loss function to preserve the structural integrity of predicted masks, thereby enhancing the coherence and precision of object boundaries. The extensive experimental results on multiple AVS datasets show that our proposed method outperforms current advanced AVS models and approaches from other tasks in terms of the mean Intersection over Union (mIoU) and F-score metrics.