检索结果-内蒙古大学图书馆

Enhancing Sound Source Localization via False Negative Elimination

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024年第12期46卷 10499-10514页

作者： Song, Zengjie Zhang, Jiangshe Wang, Yuxi Fan, Junsong Zhang, Zhaoxiang Xi An Jiao Tong Univ Sch Math & Stat Xian 710049 Peoples R China Chinese Acad Sci Hong Kong Inst Sci & Innovat Ctr Artificial Intelligence & Robot Hong Kong Peoples R China Chinese Acad Sci Inst Automat New Lab Pattern Recognit State Key Lab Multimodal Artificial Intelligence Beijing 100190 Peoples R China Univ Chinese Acad Sci Beijing 100049 Peoples R China

Sound source localization aims to localize objects emitting the sound in visual scenes. Recent works obtaining impressive results typically rely on contrastive learning. However, the common practice of randomly sampling negatives in prior arts can lead to the false negative issue, where the sounds semantically similar to visual instance are sampled as negatives and incorrectly pushed away from the visual anchor/query. As a result, this misalignment of audio and visual features could yield inferior performance. To address this issue, we propose a novel audio-visual learning framework which is instantiated with two individual learning schemes: self-supervised predictive learning (SSPL) and semantic-aware contrastive learning (SACL). SSPL explores image-audio positive pairs alone to discover semantically coherent similarities between audio and visual features, while a predictive coding module for feature alignment is introduced to facilitate the positive-only learning. In this regard SSPL acts as a negative-free method to eliminate false negatives. By contrast, SACL is designed to compact visual features and remove false negatives, providing reliable visual anchor and audio negatives for contrast. Different from SSPL, SACL releases the potential of audio-visual contrastive learning, offering an effective alternative to achieve the same goal. Comprehensive experiments demonstrate the superiority of our approach over the state-of-the-arts. Furthermore, we highlight the versatility of the learned representation by extending the approach to audio-visual event classification and object detection tasks.

关键词： visualization Contrastive learning Location awareness Task analysis Feature extraction Representation learning Semantics audio-visual learning contrastive learning feature alignment self-supervised learning sound source localization

来源：评论

学校读者我要写书评

暂无评论

UAVM: Towards Unifying audio and visual Models

引用

IEEE SIGNAL PROCESSING LETTERS 2022年 29卷 2437-2441页

作者： Gong, Yuan Liu, Alexander H. Rouditchenko, Andrew Glass, James MIT Comp Sci & Artificial Intelligence Lab Cambridge MA 02139 USA

Conventional audio-visual models have independent audio and video branches. In this work, we unify the audio and visual branches by designing a Unified audio-visual Model (UAVM). The UAVM achieves a new state-of-the-art audio-visual event classification accuracy of 65.8% on VGGSound. More interestingly, we also find a few intriguing properties of UAVM that the modality-independent counterparts do not have. Code at ***/yuangongnd/uavm.

关键词： Terms & mdash audio-visual learning unified model

来源：评论

学校读者我要写书评

暂无评论

Deep Multi-biometric Fusion for audio-visual User Re-Identification and Verification 8th

Deep Multi-biometric Fusion for Audio-Visual User Re-Identif...

引用

8th International Conference on Pattern Recognition Applications and Methods (ICPRAM)

作者： Marras, Mirko Marin-Reyes, Pedro A. Lorenzo-Navarro, Javier Castrillon-Santana, Modesto Fenu, Gianni Univ Cagliari Dept Math & Comp Sci V Osped 72 I-09124 Cagliari Italy Univ Las Palmas Gran Canaria Inst Univ Sistemas Inteligentes & Aplicac Numer I Campus Univ Tafira Las Palmas Gran Canaria 35017 Spain

ISBN: (纸本)9783030400149;9783030400132

From border controls to personal devices, from online exam proctoring to human-robot interaction, biometric technologies are empowering individuals and organizations with convenient and secure authentication and identification services. However, most biometric systems leverage only a single modality, and may face challenges related to acquisition distance, environmental conditions, data quality, and computational resources. Combining evidence from multiple sources at a certain level (e.g., sensor, feature, score, or decision) of the recognition pipeline may mitigate some limitations of the common uni-biometric systems. Such a fusion has been rarely investigated at intermediate level, i.e., when uni-biometric model parameters are jointly optimized during training. In this chapter, we propose a multi-biometric model training strategy that digests face and voice traits in parallel, and we explore how it helps to improve recognition performance in re-identification and verification scenarios. To this end, we design a neural architecture for jointly embedding face and voice data, and we experiment with several training losses and audio-visual datasets. The idea is to exploit the relation between voice characteristics and facial morphology, so that face and voice uni-biometric models help each other to recognize people when trained jointly. Extensive experiments on four real-world datasets show that the biometric feature representation of a uni-biometric model jointly trained performs better than the one computed by the same unibiometric model trained alone. Moreover, the recognition results are further improved by embedding face and voice data into a single shared representation of the two modalities. The proposed fusion strategy generalizes well on unseen and unheard users, and should be considered as a feasible solution that improves model performance. We expect that this chapter will support the biometric community to shape the research on deep audio-visual fusion in

关键词： Multi-biometric system Cross-modal biometrics Deep biometric fusion audio-visual learning Verification Re-identification

来源：评论

学校读者我要写书评

暂无评论

visually-Aware audio Captioning With Adaptive audio-visual Attention 24

Visually-Aware Audio Captioning With Adaptive Audio-Visual A...

引用

Interspeech Conference

作者： Liu, Xubo Huang, Qiushi Mei, Xinhao Liu, Haohe Kong, Qiuqiang Sun, Jianyuan Li, Shengchen Ko, Tom Zhang, Yu Tang, Lilian H. Plumbley, Mark D. Kilic, Volkan Wang, Wenwu Univ Surrey Guildford Surrey England ByteDance Beijing Peoples R China Xian Jiaotong Liverpool Univ Xian Peoples R China Southern Univ Sci & Technol Shenzhen Peoples R China Izmir Katip Celebi Univ Izmir Turkiye

audio captioning aims to generate text descriptions of audio clips. In the real world, many objects produce similar sounds. How to accurately recognize ambiguous sounds is a major challenge for audio captioning. In this work, inspired by inherent human multimodal perception, we propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to extract video features and incorporate the visual features into an audio captioning system. Furthermore, to better exploit complementary audio-visual contexts, we propose an audio-visual attention mechanism that adaptively integrates audio and visual context and removes the redundant information in the latent space. Experimental results on audioCaps, the largest audio captioning dataset, show that our proposed method achieves state-of-the-art results on machine translation metrics.

关键词： audio captioning audio-visual learning attention mechanism multimodal learning

来源：评论

学校读者我要写书评

暂无评论

DEEP VIDEO INPAINTING GUIDED BY audio-visual SELF-SUPERVISION 47

DEEP VIDEO INPAINTING GUIDED BY AUDIO-VISUAL SELF-SUPERVISIO...

引用

47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

作者： Kim, Kyuyeon Jung, Junsik Kim, Woo Jae Yoon, Sung-Eui Korea Adv Inst Sci & Technol KAIST Sch Comp Daejeon South Korea

ISBN: (纸本)9781665405409

Humans can easily imagine a scene from auditory information based on their prior knowledge of audio-visual events. In this paper, we mimic this innate human ability in deep learning models to improve the quality of video inpainting. To implement the prior knowledge, we first train the audio-visual network, which learns the correspondence between auditory and visual information. Then, the audiovisual network is employed as a guider that conveys the prior knowledge of audio-visual correspondence to the video inpainting network. This prior knowledge is transferred through our proposed two novel losses: audio-visual attention loss and audio-visual pseudoclass consistency loss. These two losses further improve the performance of the video inpainting by encouraging the inpainting result to have a high correspondence to its synchronized audio. Experimental results demonstrate that our proposed method can restore a wider domain of video scenes and is particularly effective when the sounding object in the scene is partially blinded.

关键词： audio-visual learning audio-visual correspondence audio-visual network deep video inpainting

来源：评论

学校读者我要写书评

暂无评论

Unraveling Instance Associations: A Closer Look for audio-visual Segmentation

Unraveling Instance Associations: A Closer Look for Audio-Vi...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Chen, Yuanhong Liu, Yuyuan Wang, Hu Liu, Fengbei Wang, Chong Frazer, Helen Carneiro, Gustavo Univ Adelaide Australian Inst Machine Learning Adelaide SA Australia St Vincents Hosp Melbourne Melbourne Vic Australia Univ Surrey Ctr Vis Speech & Signal Proc Guildford Surrey England

ISBN: (纸本)9798350353006

audio-visual segmentation (AVS) is a challenging task that involves accurately segmenting sounding objects based on audio-visual cues. The effectiveness of audio-visual learning critically depends on achieving accurate cross-modal alignment between sound and visual objects. Successful audio-visual learning requires two essential components: 1) a challenging dataset with high-quality pixel-level multi-class annotated images associated with audio files, and 2) a model that can establish strong links between audio information and its corresponding visual object. However, these requirements are only partially addressed by current methods, with training sets containing biased audiovisual data, and models that generalise poorly beyond this biased training set. In this work, we propose a new cost-effective strategy to build challenging and relatively unbiased high-quality audio-visual segmentation benchmarks. We also propose a new informative sample mining method for audio-visual supervised contrastive learning to leverage discriminative contrastive samples to enforce cross-modal understanding. We show empirical results that demonstrate the effectiveness of our benchmark. Furthermore, experiments conducted on existing AVS datasets and on our new benchmark show that our method achieves state-of-the-art (SOTA) segmentation accuracy(1).

关键词： audio-visual learning Multi-modal learning

来源：评论

学校读者我要写书评

暂无评论

PANO-ECHO: PANOramic depth prediction enhancement with ECHO features 2

PANO-ECHO: PANOramic depth prediction enhancement with ECHO ...

引用

2nd IEEE Conference on Artificial Intelligence (CAI)

作者： Liu, Xiaohu Brunetto, Amandine Hornauer, Sascha Moutarde, Fabien Lu, Jialiang Shanghai Jiao Tong Univ SJTU Paris Elite Inst Technol Shanghai Peoples R China PSL Univ Ctr Robot MINES Paris Paris France

ISBN: (纸本)9798350354102;9798350354096

Panoramic depth estimation gains importance with more 360 degrees images being widely available. However, traditional mono-to-depth approaches, optimized for a limited field of view, show subpar performance when naively adapted. Methods tailored to process panoramic input improve predictions but can not overcome ambiguous visual information and scale-uncertainty inherent to the task. In this paper we show the benefits of leveraging sound for improved panoramic depth estimation. Specifically, we harness audible echoes from emitted chirps as they contain rich geometric and material cues about the surrounding environment. We show that these auditory cues can enhance a state-of-the-art panoramic depth prediction framework. By integrating sound information, we improve this vision-only baseline by approximate to 12%. Our approach requires minimal modifications to the underlying architecture, making it easily applicable to other baseline models. We validate its efficacy on the Matterport3D and Replica datasets, demonstrating remarkable improvements in depth estimation accuracy. Our code is available here: https: //***/peter12398/PANO-ECHO

关键词： audio-visual learning panoramic depth estimation multi-modal fusion

来源：评论

学校读者我要写书评

暂无评论

Integrating audio-visual Contexts with Refinement for Segmentation 33rd

Integrating Audio-Visual Contexts with Refinement for Segmen...

引用

33rd International Conference on Artificial Neural Networks and Machine learning (ICANN)

作者： Geng, Qingwei Gu, Xiaodong Fudan Univ Dept Elect Engn Shanghai 200438 Peoples R China

ISBN: (纸本)9783031723377;9783031723384

A more fine-grained video spatial localization task audio visual segmentation(AVS) has recently been proposed, which aims to generate the masks of the sounding objects that sound in the given videos. In this paper, we propose a novel network for AVS. Specifically, to capture comprehensive global auditory semantic information and facilitate its interaction with visual frames, our model incorporates a transformer-based audio-visual context encoder, which is designed to generate a pixel-level score map enriched with auditory contextualized information for the decoder. In addition, to address challenges of the vague boundaries of the sounding object in videos, we introduce a refinement module as well as structural similarity(SSIM) loss to enhance accurate boundary predictions. Extensive experiments on the AVSBench dataset show that our proposed method surpasses the baseline AVSBench as well as some advanced methods from other tasks.

关键词： audio-visual learning audio visual Segmentation Multi-Modal learning

来源：评论

学校读者我要写书评

暂无评论

Masked Lip-Sync Prediction by audio-visual Contextual Exploitation in Transformers 22

Masked Lip-Sync Prediction by Audio-Visual Contextual Exploi...

引用

SIGGRAPH Asia Conference

作者： Sun, Yasheng Zhou, Hang Wang, Kaisiyuan Wu, Qianyi Hong, Zhibin Liu, Jingtuo Ding, Errui Wang, Jingdong Liu, Ziwei Koike, Hideki Tokyo Inst Technol Tokyo Japan Baidu Inc Shanghai Peoples R China Univ Sydney Sydney NSW Australia Monash Univ Melbourne Vic Australia Baidu Inc Shenzhen Peoples R China Baidu Inc Beijing Peoples R China Nanyang Technol Univ Singapore Singapore

ISBN: (纸本)9781450394703

Previous studies have explored generating accurately lip-synced talking faces for arbitrary targets given audio conditions. However, most of them deform or generate the whole facial area, leading to non-realistic results. In this work, we delve into the formulation of altering only the mouth shapes of the target person. This requires masking a large percentage of the original image and seamlessly inpainting it with the aid of audio and reference frames. To this end, we propose the audio-visual Context-Aware Transformer (AV-CAT) framework, which produces accurate lip-sync with photo-realistic quality by predicting the masked mouth shapes. Our key insight is to exploit desired contextual information provided in audio and visual modalities thoroughly with delicately designed Transformers. Specifically, we propose a convolution-Transformer hybrid backbone and design an attention-based fusion strategy for filling the masked parts. It uniformly attends to the textural information on the unmasked regions and the reference frame. Then the semantic audio information is involved in enhancing the self-attention computation. Additionally, a refinement network with audio injection improves both image and lip-sync quality. Extensive experiments validate that our model can generate high-fidelity lip-synced results for arbitrary subjects.

关键词： Lip-Sync Transformer audio-visual learning

来源：评论

学校读者我要写书评

暂无评论

Extreme-scale Talking-Face Video Upsampling with audio-visual Priors 22

Extreme-scale Talking-Face Video Upsampling with Audio-Visua...

引用

30th ACM International Conference on Multimedia (MM)

作者： Hegde, Sindhu B. Mukhopadhyay, Rudrabha Namboodiri, Vinay P. Jawahar, C. V. Int Inst Informat Technol Hyderabad India Univ Bath Bath Avon England

ISBN: (纸本)9781450392037

In this paper, we explore an interesting question of what can be obtained from an 8 x 8 pixel video sequence. Surprisingly, it turns out to be quite a lot. We show that when we process this 8 x 8 video with the right set of audio and image priors, we can obtain a full-length, 256 x 256 video. We achieve this 32x scaling of an extremely low-resolution input using our novel audio-visual upsampling network. The audio prior helps to recover the elemental facial details and precise lip shapes and a single high-resolution target identity image prior provides us with rich appearance details. Our approach is an end-to-end multi-stage framework. The first stage produces a coarse intermediate output video that can be then used to animate single target identity image and generate realistic, accurate and high-quality outputs. Our approach is simple and performs exceedingly well (an 8x improvement in FID score) compared to previous super-resolution methods. We also extend our model to talking-face video compression, and show that we obtain a 3.5x improvement in terms of bits/pixel over the previous state-of-the-art. The results from our network are thoroughly analyzed through extensive ablation experiments (in the paper and supplementary material). We also provide the demo video along with code and models on our website(1).

关键词： video upsampling video super-resolution video compression talking-face videos audio-visual learning

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：