MulDeF: A Model-Agnostic Debiasing Framework for Robust Multimodal Sentiment Analysis

作     者：Huan, Ruohong Zhong, Guowei Chen, Peng Liang, Ronghua 

作者机构：Zhejiang Univ Technol Coll Comp Sci & Technol Hangzhou 310023 Peoples R China 

出 版 物：《IEEE TRANSACTIONS ON MULTIMEDIA》 (IEEE Trans Multimedia)

年 卷 期：2025年第27卷

页      面：2304-2319页

核心收录：

学科分类：0810[工学-信息与通信工程] 0808[工学-电气工程] 08[工学] 0835[工学-软件工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：National Natural Science Foundation of China [62276237, 62036009, 62432014] Basic Public Welfare Research Program of Zhejiang Province [LTGY23F020006] Zhejiang Provincial Natural Science Foundation of China [LDT23F0202, LDT23F02021F02] 

主　　题：Visualization Training Sentiment analysis Noise Correlation Motion pictures Feature extraction Cognition Robustness Neural networks Causal inference causal intervention counterfactual reasoning multimodal sentiment analysis out-of-distribution 

摘      要：In recent years, multimodal sentiment analysis (MSA) has gained prominence with the proliferation of social media. However, prior studies have often disregarded the possibility of spurious correlations between multimodal data and sentiment labels. Neglecting these factors often results in significant performance degradation, hampering the model s ability to generalize in out-of-distribution (OOD) scenarios. To gain a comprehensive understanding of multimodal knowledge and enhance the model s generalization across diverse distribution scenarios, we present the Multimodal Debiasing Framework (MulDeF). This model-agnostic framework addresses label bias through causal intervention and tackles multimodal biases using counterfactual reasoning. During the training phase, MulDeF rectifies multimodal representations through frontdoor adjustment in causal intervention, effectively eliminating label bias. In order to model conditional expectation calculations within the context of frontdoor adjustment, we introduce multimodal causal attention (MCA). In the inference phase, it employs counterfactual reasoning to eliminate multimodal biases. To further refine our debiasing strategy, we categorize multimodal biases into two distinct types: nonverbal bias and verbal bias. Nonverbal bias is addressed at the utterance level, involving the establishment of unimodal models for audio and visual modalities to estimate their biases concerning sentiment labels. Conversely, verbal bias mitigation occurs at the word level. Here, we mask harmless words to generate corresponding counterfactual texts, which are then assessed by the text model to identify word-level bias. Experimental results validate the effectiveness of MulDeF, showcasing its superior performance in OOD settings compared to state-of-the-art methods, while also achieving competitive results in independent and identically distributed (IID) settings.