PromptFusion:Harmonized Semantic Prompt Learning for Infrared and Visible Image Fusion

作     者：Jinyuan Liu Xingyuan Li Zirui Wang Zhiying Jiang Wei Zhong Wei Fan Bin Xu 

作者机构：IEEE the School of Software Technology Dalian University of Technology the School of Mechanical Engineering Beijing Institute of Technology 

出 版 物：《IEEE/CAA Journal of Automatica Sinica》 (自动化学报(英文版))

年 卷 期：2025年第12卷第3期

页      面：502-515页

核心收录：

学科分类：080901[工学-物理电子学] 0809[工学-电子科学与技术（可授工学、理学学位）] 08[工学] 080401[工学-精密仪器及机械] 080203[工学-机械设计及理论] 0804[工学-仪器科学与技术] 0802[工学-机械工程] 0803[工学-光学工程] 

基　　金：partially supported by China Postdoctoral Science Foundation (2023M730741) the National Natural Science Foundation of China (U22B2052, 52102432, 52202452, 62372080, 62302078) 

主　　题：Bi-level optimization image fusion infrared and visible image prompt learning 

摘      要：The goal of infrared and visible image fusion(IVIF)is to integrate the unique advantages of both modalities to achieve a more comprehensive understanding of a scene. However, existing methods struggle to effectively handle modal disparities,resulting in visual degradation of the details and prominent targets of the fused images. To address these challenges, we introduce Prompt Fusion, a prompt-based approach that harmoniously combines multi-modality images under the guidance of semantic prompts. Firstly, to better characterize the features of different modalities, a contourlet autoencoder is designed to separate and extract the high-/low-frequency components of different modalities, thereby improving the extraction of fine details and textures. We also introduce a prompt learning mechanism using positive and negative prompts, leveraging Vision-Language Models to improve the fusion model s understanding and identification of targets in multi-modality images, leading to improved performance in downstream tasks. Furthermore, we employ bi-level asymptotic convergence optimization. This approach simplifies the intricate non-singleton non-convex bi-level problem into a series of convergent and differentiable single optimization problems that can be effectively resolved through gradient *** approach advances the state-of-the-art, delivering superior fusion quality and boosting the performance of related downstream tasks. Project page: https://***/hey-it-s-me/PromptFusion.