检索结果-内蒙古大学图书馆

TransFuse: A Unified Transformer-based Image Fusion Framework using Self-supervised Learning

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Qu, Linhao Liu, Shaolei Wang, Manning Li, Shiman Yin, Siqi Qiao, Qin Song, Zhijian Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention Digital Medical Research Center School of Basic Medical Science Fudan University Shanghai200032 China

Image fusion is a technique to integrate information from multiple source images with complementary information to improve the richness of a single image. Due to insufficient task-specific training data and corresponding ground truth, most existing end-to-end image fusion methods easily fall into overfitting or tedious parameter optimization processes. Two-stage methods avoid the need of large amount of task-specific training data by training encoder-decoder network on large natural image datasets and utilizing the extracted features for fusion, but the domain gap between natural images and different fusion tasks results in limited performance. In this study, we design a novel encoder-decoder based image fusion framework and propose a destruction-reconstruction based self-supervised training scheme to encourage the network to learn task-specific features. Specifically, we propose three destruction-reconstruction self-supervised auxiliary tasks for multi-modal image fusion, multi-exposure image fusion and multi-focus image fusion based on pixel intensity non-linear transformation, brightness transformation and noise transformation, respectively. In order to encourage different fusion tasks to promote each other and increase the generalizability of the trained network, we integrate the three self-supervised auxiliary tasks by randomly choosing one of them to destroy a natural image in model training. In addition, we design a new encoder that combines CNN and Transformer for feature extraction, so that the trained model can exploit both local and global information. Extensive experiments on multi-modal image fusion, multi-exposure image fusion and multi-focus image fusion tasks demonstrate that our proposed method achieves the state-of-the-art performance in both subjective and objective evaluations. The code will be publicly available soon. Copyright © 2022, The Authors. All rights reserved.

关键词： Deep learning

Robust point cloud registration framework based on deep graph matching

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Fu, Kexue Liu, Shaolei Luo, Xiaoyuan Wang, Manning Digital Medical Research Center School of Basic Medical Science Fudan University Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention China

3D point cloud registration is a fundamental problem in computer vision and robotics. There has been extensive research in this area, but existing methods meet great challenges in situations with a large proportion of outliers and time constraints, but without good transformation initialization. Recently, a series of learning-based algorithms have been introduced and show advantages in speed. Many of them are based on correspondences between the two point clouds, so they do not rely on transformation initialization. However, these learning-based methods are sensitive to outliers, which lead to more incorrect correspondences. In this paper, we propose a novel deep graph matching-based framework for point cloud registration. Specifically, we first transform point clouds into graphs and extract deep features for each point. Then, we develop a module based on deep graph matching to calculate a soft correspondence matrix. By using graph matching, not only the local geometry of each point but also its structure and topology in a larger range are considered in establishing correspondences, so that more correct correspondences are found. We train the network with a loss directly defined on the correspondences, and in the test stage the soft correspondences are transformed into hard one-to-one correspondences so that registration can be performed by singular value decomposition. Furthermore, we introduce a transformer-based method to generate edges for graph construction, which further improves the quality of the correspondences. Extensive experiments on registering clean, noisy, partial-to-partial and unseen category point clouds show that the proposed method achieves state-of-the-art performance. The code will be made publicly available at https://***/fukexue/RGM. Copyright © 2021, The Authors. All rights reserved.

关键词： Surface measurement

Towards label-efficient Automatic Diagnosis and Analysis: A Comprehensive Survey of Advanced Deep Learning-based Weakly-supervised, Semi-supervised and Self-supervised Techniques in Histopathological Image Analysis

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Qu, Linhao Liu, Siyu Liu, Xiaoyu Wang, Manning Song, Zhijian Digital Medical Research Center School of Basic Medical Science Fudan University Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention Shanghai200032 China

Histopathological images contain abundant phenotypic information and pathological patterns, which are the gold standards for disease diagnosis and essential for the prediction of patient prognosis and treatment outcome. In recent years, computer-automated analysis techniques for histopathological images have been urgently required in clinical practice, and deep learning methods represented by convolutional neural networks have gradually become the mainstream in the field of digital pathology. However, obtaining large numbers of fine-grained annotated data in this field is a very expensive and difficult task, which hinders the further development of traditional supervised algorithms based on large numbers of annotated data. More recent studies have started to liberate from the traditional supervised paradigm, and the most representative ones are the studies on weakly supervised learning paradigm based on weak annotation, semi-supervised learning paradigm based on limited annotation, and self-supervised learning paradigm based on pathological image representation learning. These new methods have led a new wave of automatic pathological image diagnosis and analysis targeted at annotation efficiency. With a survey of over 130 papers, we present a comprehensive and systematic review of the latest studies on weakly supervised learning, semi-supervised learning, and self-supervised learning in the field of computational pathology from both technical and methodological perspectives. Finally, we present the key challenges and future trends for these techniques. Copyright © 2022, The Authors. All rights reserved.

关键词： Surveys

Segmentation of arteriovenous malformations nidus and vessel in digital subtraction angiography images based on an iterative thresholding method

学校读者我要写书评

暂无评论

Segmentation of arteriovenous malformations nidus and vessel...

International Conference on Biomedical Engineering and Informatics (BMEI)

作者： Yuxi Lian Yuanyuan Wang Jinhua Yu Yi Guo Liang Chen Department of Electronic Engineering Fudan University Shanghai China Key laboratory of Medical Imaging Computing and-Computer Assisted Intervention of Shanghai China Department of Neurosurgery Fudan University Shanghai China

Digital subtraction angiography (DSA) plays an important role in the diagnosis and therapy of vascular diseases. Segmentation of nidus and vessel in DSA images is an essential step in the diagnosis of arteriovenous malformations (AVM). In this paper, a novel segmentation method based on the global and iterative local thresholding is proposed to segment the nidus and vessel in DSA images. Firstly, the original image is divided into proper subimages. For each subimage, Ostu's method is primarily used and pixels are classified into two groups by the threshold. Then, according to the variance of the subimage intensities, the mean or median values of two groups are calculated to sort the pixels into three classes. These three classes represent the dark AVM and vessel, the bright background and undetermined regions in the original DSA image. The first two classes are determined directly and will not be processed further. The undetermined regions are processed in the next iteration to segment tiny vessels until the thresholds between two iterations are less than a preset one. Finally, all classes are combined to create the segmentation result. We test this method on DSA images of the AVM. Experimental results show that the proposed method performs better than the other state-of-the-art methods in the segmentation of DSA images. The proposed method can identify fine and tiny vessel structures, as well as distinguish large AVM nidus in one framework.

关键词： Image segmentation Iterative methods Angiography Diseases Blood vessels Hospitals

Transfuse: A Unified Transformer-Based Image Fusion Framework Using Self-Supervised Learning

学校读者我要写书评

暂无评论

SSRN

SSRN 2022年

关键词： Deep learning

Fusionmlp: A Mlp-Based Unified Image Fusion Framework

学校读者我要写书评

暂无评论

SSRN

SSRN 2023年

作者： Liu, Shaolei Qu, Linhao Wang, Manning Song, Zhijian Digital Medical Research Center School of Basic Medical Science Fudan University Shanghai200032 China Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention China

Due to the powerful feature representation capacity, deep learning-based image fusion methods have improved the fusion results for better information integration. However, some inherent limitations in convolutional neural networks (CNN) and Transformer networks impede further performance improvement. Specifically, CNN-based methods have difficulties in mining global context information;Transformer-based methods are unable to adapt to different image sizes and the computational complexity grows exponentially with the image size. To overcome these problems, we propose the first MLPbased multi-scene image fusion framework, denoted as FusionMLP. In addition, we propose a novel loss function combining the spatial and frequency domain information for model training. Extensive experiments show that FusionMLP not only outperform SOTA multi-scene image fusion methods, but also outperform SOTA fusion algorithms for specific scenes. Codes will be released at https://***/slliuEric/FusionMLP. © 2023, The Authors. All rights reserved.

关键词： Image fusion

Fusionmlp: A Mlp-Based Unified Image Fusion Framework

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Liu, Shaolei Li, Shiman Qu, Linhao Wang, Manning Song, Zhijian Digital Medical Research Center School of Basic Medical Science Fudan University Shanghai200032 China Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention China

Due to the powerful feature representation capacity, deep learning-based image fusion methods have improved the fusion results for better information integration. However, some inherent limitations in convolutional neural networks (CNN) and Transformer networks impede further performance improvement. Specifically, CNN-based methods have difficulties in mining global context information;Transformer-based methods are unable to adapt to different image sizes and the computational complexity grows exponentially with the image size. To overcome these problems, we propose the first MLP-based multi-scene image fusion framework, denoted as FusionMLP. In addition, we propose a novel loss function combining the spatial and frequency domain information for model training. Extensive experiments show that FusionMLP not only outperform SOTA multi-scene image fusion methods, but also outperform SOTA fusion algorithms for specific scenes. © 2024, The Authors. All rights reserved.

关键词： Image fusion

The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Qu, Linhao Luo, Xiaoyuan Fu, Kexue Wang, Manning Song, Zhijian Digital Medical Research Center School of Basic Medical Science Fudan University China Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention China

This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Since a WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached as a Multiple Instance Learning (MIL) problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC is to classify both bags and instances with only a limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose a two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based on the bag features. Our approach incorporates the utilization of GPT-4 in a question- and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification. Codes will be available at https://***/miccaiif/TOP. Copyright © 2023, The Authors. All rights reserved.

关键词： Pathology

TransMEF: A transformer-based multi-exposure image fusion framework using self-supervised multi-task learning

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Qu, Linhao Liu, Shaolei Wang, Manning Song, Zhijian Digital Medical Research Center School of Basic Medical Science Fudan University Shanghai200032 China Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention China

In this paper, we propose TransMEF, a transformer-based multi-exposure image fusion framework that uses self-supervised multi-task learning. The framework is based on an encoder-decoder network, which can be trained on large natural image datasets and does not require ground truth fusion images. We design three self-supervised reconstruction tasks according to the characteristics of multi-exposure images and conduct these tasks simultaneously using multi-task learning;through this process, the network can learn the characteristics of multi-exposure images and extract more generalized features. In addition, to compensate for the defect in establishing long-range dependencies in CNN-based architectures, we design an encoder that combines a CNN module with a transformer module. This combination enables the network to focus on both local and global information. We evaluated our method and compared it to 11 competitive traditional and deep learning-based methods on the latest released multi-exposure image fusion benchmark dataset, and our method achieved the best performance in both subjective and objective evaluations. Copyright © 2021, The Authors. All rights reserved.

关键词： Image fusion