StfNet: A Two-Stream Convolutional Neural Network for Spatiotemporal Image Fusion

作     者：Liu, Xun Deng, Chenwei Chanussot, Jocelyn Hong, Danfeng Zhao, Baojun 

作者机构：Beijing Inst Technol Sch Informat & Elect Beijing Key Lab Embedded Real Time Informat Proc Beijing 100081 Peoples R China Univ Grenoble Alpes CNRS Grenoble INP GIPSA Lab F-38000 Grenoble France German Aerosp Ctr DLR Remote Sensing Technol Inst IMF D-82234 Wessling Germany TUM Signal Proc Earth Observat SiPEO D-80333 Munich Germany 

出 版 物：《IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING》 (IEEE地学与遥感汇刊)

年 卷 期：2019年第57卷第9期

页      面：6552-6564页

核心收录：

学科分类：0808[工学-电气工程] 1002[医学-临床医学] 08[工学] 0708[理学-地球物理学] 0816[工学-测绘科学与技术] 

基　　金：National Natural Science Foundation of China (NSFC) 

主　　题：Convolutional neural network spatiotemporal image fusion super-resolution temporal consistency temporal dependence (TD) 

摘      要：Spatiotemporal image fusion is considered as a promising way to provide Earth observations with both high spatial resolution and frequent coverage, and recently, learning-based solutions have been receiving broad attention. However, these algorithms treating spatiotemporal fusion as a single image super-resolution problem, generally suffers from the significant spatial information loss in coarse images, due to the large upscaling factors in real applications. To address this issue, in this paper, we exploit temporal information in fine image sequences and solve the spatiotemporal fusion problem with a two-stream convolutional neural network called StfNet. The novelty of this paper is twofold. First, considering the temporal dependence among image sequences, we incorporate the fine image acquired at the neighboring date to super-resolve the coarse image at the prediction date. In this way, our network predicts a fine image not only from the structural similarity between coarse and fine image pairs but also by exploiting abundant texture information in the available neighboring fine images. Second, instead of estimating each output fine image independently, we consider the temporal relations among time-series images and formulate a temporal constraint. This temporal constraint aiming to guarantee the uniqueness of the fusion result and encourages temporal consistent predictions in learning and thus leads to more realistic final results. We evaluate the performance of the StfNet using two actual data sets of Landsat-Moderate Resolution Imaging Spectroradiometer (MODIS) acquisitions, and both visual and quantitative evaluations demonstrate that our algorithm achieves state-of-the-art performance.