Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection

作     者：Guan, Yinghan Dai, Haoran Yu, Zekuan Wang, Shouyu Gu, Yuanjie 

作者机构：School of Electronics and Information Engineering Wuxi University Wuxi214105 China Metropolitan College Department of Computer Science Boston University Boston02215 United States Academy for Engineering and Technology Fudan University Shanghai200433 China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2023年

核心收录：

主　　题：Imaging systems 

摘      要：Multimodal fusion detection always places high demands on the imaging system and image pre-processing, while either a high-quality pre-registration system or image registration processing is costly. Unfortunately, the existing fusion methods are designed for registered source images, and the fusion of inhomogeneous features, which denotes a pair of features at the same spatial location that expresses different semantic information, cannot achieve satisfactory performance via these methods. As a result, we propose IA-VFDnet, a CNN-Transformer hybrid learning framework with a unified high-quality multimodal feature matching module (AKM) and a fusion module (WDAF), in which AKM and DWDAF work in synergy to perform high-quality infrared-aware visible fusion detection, which can be applied to smoke and wildfire detection. Furthermore, experiments on the M3FD dataset validate the superiority of the proposed method, with IA-VFDnet achieving the best detection performance than other state-of-the-art methods under conventional registered conditions. In addition, the first unregistered multimodal smoke and wildfire detection benchmark is openly available in this letter. Copyright © 2023, The Authors. All rights reserved.