As a fine-grained task in the community of multi-modalsentimentanalysis (MSA), multi- modalaspect-basedsentimentanalysis (MABSA) is challenging and has attracted numerous researchers' attention, and prominent...
详细信息
As a fine-grained task in the community of multi-modalsentimentanalysis (MSA), multi- modalaspect-basedsentimentanalysis (MABSA) is challenging and has attracted numerous researchers' attention, and prominent progress has been achieved in recent years. However, there is still a lack of effective strategies for feature alignment between different modalities, and further exploration is urgently needed. Thus, this paper proposed a novel MABSA method to enhance the sentiment feature alignment, namely Affective Knowledge-Assisted Bi-directional Learning (AKABL) networks, which learn sentiment information from different modalities through multiple perspectives. Specifically, AKABL gains the textual semantic and syntactic features through encoding text modality via pre-trained language model BERT and Syntax Parser SpaCy, respectively. And then, to strengthen the expression of sentiment information in the syntactic graph, affective knowledge SenticNet is introduced to assist AKABL in comprehending textual sentiment information. On the other side, to leverage image modality efficiently, the pre-trained model Visual Transformer (ViT) is employed to extract the necessary image features. Additionally, to integrate the obtained features, this paper utilizes the module Single modality GCN (SMGCN) to achieve the joint textual semantic and syntactic representation. And to bridge the textual and image features, the module Double modalities GCN (DMGCN) is devised and applied to extract the sentiment information from different modalities simultaneously. Besides, to bridge the alignment gap between text and image features, this paper devises a novel alignment strategy to build the relationship between these two representations, which measures that difference with the Jensen-Shannon divergence from bi-directional perspectives. It is worth noting that cross-attention and cosine distance-based similarity are also applied in the proposed AKABL. To validate the effectiveness of the pro
multi-modal aspect-based sentiment analysis (MABSA) aims to identify the sentiment polarity of aspects by incorporating visual information into text. Image and text are two types of modality information with significa...
详细信息
multi-modal aspect-based sentiment analysis (MABSA) aims to identify the sentiment polarity of aspects by incorporating visual information into text. Image and text are two types of modality information with significant modality gaps in both data form and semantic expression. Narrowing the modality gaps and feature fusion are two crucial challenges in MABSA. To address these issues, this paper introduces an aspect-enhanced alignment and fusion strategy with dual-layer contrastive learning to tackle the cross-modal fusion problem. Unlike traditional contrastive learning methods, our approach increases the number of negative samples, enabling the model to learn more discriminative features and better capture fine-grained cross-modal relationships. The proposed approach leverages overlapping aspect information as multi-modal pivots to first bridge the modality gaps and then integrate visual and text information in the multi-modal feature space, thereby improving multi-modalsentimentanalysis performance. We first introduce an aspect-guided modality alignment strategy that narrows the fundamental modality gaps between image and text using modality contrastive learning. Then, we design an aspect-oriented multi-modal fusion approach to promote cross-modal feature fusion through symmetric cross-modal interaction. Extensive experiments demonstrate that the proposed approach outperforms other state-of-the-art (SOTA) MABSA methods on three MABSA benchmark datasets. In-depth analysis further validates the effectiveness of the proposed multi-modal fusion approach for MABSA.
multi-modalaspect-oriented sentiment classification (MASC) is a fine-grain task, which aims to detect the sentiment polarity of specific aspect. However, conventional studies suffer from two issues. It is difficult t...
详细信息
暂无评论