Modality Decoupling is All You Need: A Simple Solution for Unsupervised Hyperspectral Image Fusion

作     者：Du, Songcheng Zou, Yang Wang, Zixu Li, Xingyuan Li, Ying Shen, Qiang 

作者机构：School of Computer Science Northwestern Polytechnical University China School of Software Technology Dalian University of Technology China Institute of Mathematics Physics and Computer Science Aberystwyth University United Kingdom The National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology Northwestern Polytechnical University China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Spectral resolution 

摘      要：Hyperspectral Image Fusion (HIF) aims to fuse low-resolution hyperspectral images (LR-HSIs) and high-resolution multispectral images (HR-MSIs) to reconstruct high spatial and high spectral resolution images. Current methods typically apply direct fusion from the two modalities without valid supervision, failing to fully perceive the deep modality-complementary information and hence, resulting in a superficial understanding of inter-modality connections. To bridge this gap, we propose a simple and effective solution for unsupervised HIF with an assumption that modality decoupling is essential for HIF. We introduce the modality clustering loss that ensures clear guidance of the modality, decoupling towards modality-shared features while steering clear of modality-complementary ones. Also, we propose an end-to-end Modality-Decoupled Spatial-Spectral Fusion (MossFuse) framework that decouples shared and complementary information across modalities and aggregates a concise representation of the LR-HSI and HR-MSI to reduce the modality redundancy. Systematic experiments over multiple datasets demonstrate that our simple and effective approach consistently outperforms the existing HIF methods while requiring considerably fewer parameters with reduced inference time. Copyright © 2024, The Authors. All rights reserved.