Enhancing Micro Gesture Recognition for Emotion Understanding via Context-Aware Visual-Text Contrastive Learning

作     者：Li, Deng Xing, Bohao Liu, Xin 

作者机构：Lappeenranta Lahti Univ Technol LUT Comp Vis & Pattern Recognit Lab Lappeenranta 53850 Finland Tianjin Univ Sch Elect & Informat Engn Tianjin 300072 Peoples R China Lappeenranta Lahti Univ Technol LUT Comp Vis & Pattern Recognit Lab Lappeenranta 53850 Finland 

出 版 物：《IEEE SIGNAL PROCESSING LETTERS》 (IEEE Signal Process Lett)

年 卷 期：2024年第31卷

页      面：1309-1313页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 

主　　题：Emotion understanding micro gesture recognition multimodality learning 

摘      要：Psychological studies have shown that Micro Gestures (MG) are closely linked to human emotions. MG-based emotion understanding has attracted much attention because it allows for emotion understanding through nonverbal body gestures without relying on identity information (e.g., facial and electrocardiogram data). Therefore, it is essential to recognize MG effectively for advanced emotion understanding. However, existing Micro Gesture Recognition (MGR) methods utilize only a single modality (e.g., RGB or skeleton) while overlooking crucial textual information. In this letter, we propose a simple but effective visual-text contrastive learning solution that utilizes text information for MGR. In addition, instead of using handcrafted prompts for visual-text contrastive learning, we propose a novel module called Adaptive prompting to generate context-aware prompts. The experimental results show that the proposed method achieves state-of-the-art performance on two public datasets. Furthermore, based on an empirical study utilizing the results of MGR for emotion understanding, we demonstrate that using the textual results of MGR significantly improves performance by 6%+ compared to directly using video as input.