Image sentiment analysis has received a lot of attention due to the fact that users prefer to express their private emotions using images on social platforms. Researchers strive to obtain more discriminative features ...
详细信息
Image sentiment analysis has received a lot of attention due to the fact that users prefer to express their private emotions using images on social platforms. Researchers strive to obtain more discriminative features to characterize these images by designing diverse networks, such as convolutional neural networks (CNNs) and Transformer. This method requires prior knowledge in advance and its feature fusion ignores the correlation between heterogeneous networks. An efficient but effective integration of heterogeneous Transformer and CNN deserves further study. To this end, we propose a novel model called closed-loop and knowledge iterative (ClKI) via self-distillation for image sentiment analysis. First, CNN features at different scales are extracted to capture the complementary multi-scale information, and then input into a Transformer stacked by multiple encoders with the ProbSparse self-attention to acquire more discriminative representations. Second, we adopt the true labels and deepest classification results jointly guide the training procedure of shallow modules via self-distillation. Thus, the ClKI model achieves an efficient interaction between the deep and shallow modules, providing sufficient knowledge for image sentiment analysis. Our model achieves 83.35%, 65.64%, and 91.01% accuracy on the FI, Emotion 6, and Twitter I datasets respectively, which outperforms state-of-the-art methods. Additional experiments further validate that ClKI can resolve the data imbalance problem to some degree and it has good generalization ability. Notably, ClKI is plug-and-play, promoting its practicality. Briefly, this study provides an innovative approach in image sentiment analysis, offering valuable insights for the integration of heterogeneous networks as well as knowledgeiterative method.
暂无评论