检索结果-内蒙古大学图书馆

20th International Symposium on Bioinformatics Research and Applications (ISBRA)

作者： Cheng, Jianhong Feng, Rui Li, Jinyang Xu, Jun Inst Guizhou Aerosp Measuring & Testing Technol Guiyang 550009 Peoples R China

ISBN: (纸本)9789819751303;9789819751310

Brain tumor segmentation in multimodal medical imaging poses challenges due to tumor heterogeneity and variability in imaging modalities. However, in clinical practice, the frequent absence of one or more modalities often renders multimodal glioma segmentation models either inapplicable or with poor segmentation accuracy. To address this challenge, we propose an incomplete multimodal learning framework with Modality-Aware Feature Interaction (IML-MAFI) for brain tumor segmentation. Specifically, we devise a modality-aware feature interaction (MAFI) mechanism incorporating a modality missing status code to dynamically facilitate feature interaction across modalities in diverse modality absence scenarios. MAFI is a straightforward yet potent module, leveraging graph structure and attention mechanisms to acquire and interact with complementary features among graph nodes. Concurrently, the novel modality missing status code, which signals the presence or absence of each modality, steers MAFI towards learning adaptive complementary information among nodes in various missing modality scenarios. By integrating MAFI with the modality missing status code into the encoder stage of a U-shaped architecture, we have devised a unified yet stable convolutional network to effectively segment tumors under various potential missing modality scenarios. Experimental findings demonstrate that our approach achieves state-of-the-art performance in brain tumor segmentation task involving missing modalities.

关键词： Brain tumor segmentation incomplete multimodal learning Feature interaction

来源：评论

学校读者我要写书评

暂无评论

incomplete multimodal learning for Visual Acuity Prediction After Cataract Surgery Using Masked Self-Attention 26th

Incomplete Multimodal Learning for Visual Acuity Prediction ...

引用

26th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)

作者： Zhou, Qian Zou, Hua Jiang, Haifeng Wang, Yong Wuhan Univ Sch Comp Sci Wuhan Peoples R China Wuhan Univ Aier Eye Hosp Wuhan Peoples R China

ISBN: (纸本)9783031439896;9783031439902

As the primary treatment option for cataracts, it is estimated that millions of cataract surgeries are performed each year globally. Predicting the Best Corrected Visual Acuity (BCVA) in cataract patients is crucial before surgeries to avoid medical disputes. However, accurate prediction remains a challenge in clinical practice. Traditional methods based on patient characteristics and surgical parameters have limited accuracy and often underestimate postoperative visual acuity. In this paper, we propose a novel framework for predicting visual acuity after cataract surgery using masked self-attention. Especially different from existing methods, which are based on monomodal data, our proposed method takes preoperative images and patient demographic data as input to leverage multimodal information. Furthermore, we expand our method to a more complex and challenging clinical scenario, i.e., the incomplete multimodal data. Firstly, we apply efficient Transformers to extract modality-specific features. Then, an attentional fusion network is utilized to fuse the multimodal information. To address the modality-missing problem, an attention mask mechanism is proposed to improve the robustness. We evaluate our method on a collected dataset of 1960 patients who underwent cataract surgery and compare its performance with other state-of-the-art approaches. The results show that our proposed method outperforms other methods and achieves a mean absolute error of 0.122 logMAR. The percentages of the prediction errors within +/- 0.10 logMAR are 94.3%. Besides, extensive experiments are conducted to investigate the effectiveness of each component in predicting visual acuity. Codes will be available at https://***/liyiersan/MSA.

关键词： incomplete multimodal learning Visual Acuity Prediction Self-Attention

来源：评论

学校读者我要写书评

暂无评论

SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for incomplete multimodal learning in conversational emotion recognition

引用

KNOWLEDGE-BASED SYSTEMS 2025年 309卷

作者： Fu, Fangze Ai, Wei Yang, Fan Shou, Yuntao Meng, Tao Li, Keqin Cent South Univ Forestry & Technol Coll Comp & Math Changsha 410004 Hunan Peoples R China SUNY Coll New Paltz Dept Comp Sci New Paltz NY 12561 USA

multimodal Emotion Recognition in Conversations (MERC) aims to classify utterance emotions using textual, auditory, and visual modal features. Most existing MERC methods assume each utterance has complete modalities, overlooking the common issue of incomplete modalities in real-world scenarios. Recently, graph neural networks (GNNs) have achieved notable results in incomplete multimodal Emotion Recognition in Conversations (IMERC). However, traditional GNNs focus on binary relationships between nodes, limiting their ability to capture more complex, higher-order information. Moreover, repeated message passing can cause over-smoothing, reducing their capacity to preserve essential high-frequency details. To address these issues, we propose a Spectral Domain Reconstruction Graph Neural Network (SDR-GNN) for incomplete multimodal learning in conversational emotion recognition. SDR-GNN constructs an utterance semantic interaction graph using a sliding window based on both speaker and context relationships to model emotional dependencies. To capture higher-order and high-frequency information, SDR-GNN utilizes weighted relationship aggregation, ensuring consistent semantic feature extraction across utterances. Additionally, it performs multi-frequency aggregation in the spectral domain, enabling efficient recovery of incomplete modalities by extracting both high- and low-frequency information. Finally, multi-head attention is applied to fuse and optimize features for emotion recognition. Extensive experiments on various real-world datasets demonstrate that our approach is effective in incomplete multimodal learning and outperforms current state-of-the-art methods.

关键词： incomplete multimodal learning Conversational emotion recognition multimodal fusion Spectral domain reconstruction

来源：评论

学校读者我要写书评

暂无评论

Diffusion-driven incomplete multimodal learning for Air Quality Prediction

引用

ACM TRANSACTIONS ON INTERNET OF THINGS 2025年第1期6卷 1-24页

作者： Fan, Jinxiao Qi, Mengshi Liu, Liang Ma, Huadong Beijing Univ Posts & Telecommun Comp Sci Beijing Peoples R China

Predicting air quality using multimodal data is crucial to comprehensively capture the diverse factors influencing atmospheric conditions. Therefore, this study introduces a multimodal learning framework that integrates outdoor images with traditional ground-based observations to improve the accuracy and reliability of air quality predictions. However, aligning and fusing these heterogeneous data sources poses a formidable challenge, further exacerbated by pervasive data incompleteness issues in practice. In this article, we propose a novel incomplete multimodal learning approach (iMMAir) to recovery missing data for robust air quality prediction. Specifically, we first design a shallow feature extractor to capture modal-specific features within the latent representation space. Then we develop a conditional diffusion-driven recovery module to mitigate the distribution gap between the recovered and true data. This module further incorporates two conditional constraints of temporal correlation and semantic consistency for effective modal completion. Finally, we reconstruct incomplete modalities and fuse available data using a multimodal transformer network to predict the air quality. To alleviate the modality imbalance problem, we employ an adaptive gradient modulation strategy to adjust the optimization of each modality. Experimental results demonstrate that iMMAir significantly reduces prediction errors, outperforming baseline models by an average of 5.6% and 2.5% in air quality regression and classification tasks. Our source code and data are available at https://***/pestasu/IMMAir.

关键词： incomplete multimodal learning air quality prediction diffusion model

来源：评论

学校读者我要写书评

暂无评论

A systematic literature review on incomplete multimodal learning: techniques and challenges

引用

SYSTEMS SCIENCE & CONTROL ENGINEERING 2025年第1期13卷

作者： Zhan, Yifan Yang, Rui You, Junxian Huang, Mengjie Liu, Weibo Liu, Xiaohui Xian Jiaotong Liverpool Univ Xian Peoples R China Univ Liverpool Sch Elect Engn Elect & Comp Sci Liverpool England Xian Jiaotong Liverpool Univ Design Sch Suzhou Peoples R China Brunel Univ Uxbridge England

Recently, machine learning technologies have been successfully applied across various fields. However, most existing machine learning models rely on unimodal data for information inference, which hinders their ability to generalize to complex application scenarios. This limitation has resulted in the development of multimodal learning, a field that integrates information from different modalities to enhance models' capabilities. However, data often suffers from missing or incomplete modalities in practical applications. This necessitates that models maintain robustness and effectively infer complete information in the presence of missing modalities. The emerging research direction of incomplete multimodal learning (IML) aims to facilitate effective learning from incomplete multimodal training sets, ensuring that models can dynamically and robustly address new instances with arbitrary missing modalities during the testing phase. This paper offers a comprehensive review of methods based on IML. It categorizes existing approaches based on their information sources into two main types: based on internal information and external information methods. These categories are further subdivided into data-based, feature-based, knowledge transfer-based, graph knowledge enhancement-based, and human-in-the-loop-based methods. The paper conducts comparative analyses from two perspectives: comparisons among similar methods and comparisons among different types of methods. Finally, it offers insights into the research trends in IML.

关键词： incomplete multimodal learning multimodal learning modality missing

来源：评论

学校读者我要写书评

暂无评论

GCNet: Graph Completion Network for incomplete multimodal learning in Conversation

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023年第7期45卷 8419-8432页

作者： Lian, Zheng Chen, Lan Sun, Licai Liu, Bin Tao, Jianhua Chinese Acad Sci Inst Automat Natl Lab Pattern Recognit Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 100049 Peoples R China Tsinghua Univ Dept Automat Beijing 100084 Peoples R China

Conversations have become a critical data format on social media platforms. Understanding conversation from emotion, content and other aspects also attracts increasing attention from researchers due to its widespread application in human-computer interaction. In real-world environments, we often encounter the problem of incomplete modalities, which has become a core issue of conversation understanding. To address this problem, researchers propose various methods. However, existing approaches are mainly designed for individual utterances rather than conversational data, which cannot fully exploit temporal and speaker information in conversations. To this end, we propose a novel framework for incomplete multimodal learning in conversations, called "Graph Complete Network (GCNet)," filling the gap of existing works. Our GCNet contains two well-designed graph neural network-based modules, "Speaker GNN" and "Temporal GNN," to capture temporal and speaker dependencies. To make full use of complete and incomplete data, we jointly optimize classification and reconstruction tasks in an end-to-end manner. To verify the effectiveness of our method, we conduct experiments on three benchmark conversational datasets. Experimental results demonstrate that our GCNet is superior to existing state-of-the-art approaches in incomplete multimodal learning.

关键词： Oral communication Correlation Data models Task analysis Feature extraction Tensors Benchmark testing Conversational data graph complete network (GCNet) incomplete multimodal learning speaker-sensitive modeling temporal-sensitive modeling

来源：评论

学校读者我要写书评

暂无评论

mmFormer: multimodal Medical Transformer for incomplete multimodal learning of Brain Tumor Segmentation 25th

mmFormer: Multimodal Medical Transformer for Incomplete Mult...

引用

25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)

作者： Zhang, Yao He, Nanjun Yang, Jiawei Li, Yuexiang Wei, Dong Huang, Yawen Zhang, Yang He, Zhiqiang Zheng, Yefeng Chinese Acad Sci Inst Comp Technol Beijing Peoples R China Univ Chinese Acad Sci Beijing Peoples R China Tencent Jarvis Lab Shenzhen Peoples R China Univ Calif Los Angeles Elect & Comp Engn Los Angeles CA USA Lenovo Res Beijing Peoples R China

ISBN: (纸本)9783031164439;9783031164422

Accurate brain tumor segmentation from Magnetic Resonance Imaging (MRI) is desirable to joint learning of multimodal images. However, in clinical practice, it is not always possible to acquire a complete set of MRIs, and the problem of missing modalities causes severe performance degradation in existing multimodal segmentation methods. In this work, we present the first attempt to exploit the Transformer for multimodal brain tumor segmentation that is robust to any combinatorial subset of available modalities. Concretely, we propose a novel multimodal Medical Transformer (mmFormer) for incomplete multimodal learning with three main components: the hybrid modality-specific encoders that bridge a convolutional encoder and an intra-modal Transformer for both local and global context modeling within each modality;an inter-modal Transformer to build and align the long-range correlations across modalities for modality-invariant features with global semantics corresponding to tumor region;a decoder that performs a progressive up-sampling and fusion with the modality-invariant features to generate robust segmentation. Besides, auxiliary regularizers are introduced in both encoder and decoder to further enhance the model's robustness to incomplete modalities. We conduct extensive experiments on the public BraTS 2018 dataset for brain tumor segmentation. The results demonstrate that the proposed mmFormer outperforms the state-of-the-art methods for incomplete multimodal brain tumor segmentation on almost all subsets of incomplete modalities, especially by an average 19.07% improvement of Dice on tumor segmentation with only one available modality. The code is available at https://***/YaoZhang93/mmFormer.

关键词： incomplete multimodal learning Brain tumor segmentation Transformer

来源：评论

学校读者我要写书评

暂无评论

GMRLNet: A Graph-Based Manifold Regularization learning Framework for Placental Insufficiency Diagnosis on incomplete multimodal Ultrasound Data

引用

IEEE TRANSACTIONS ON MEDICAL IMAGING 2023年第11期42卷 3205-3218页

作者： Jiao, Jing Sun, Hongshuang Huang, Yi Xia, Menghua Qiao, Mengyun Ren, Yunyun Wang, Yuanyuan Guo, Yi Fudan Univ Sch Informat Sci & Technol Dept Elect Engn Shanghai 200433 Peoples R China Fudan Univ Obstet & Gynecol Hosp Dept Ultrasound Shanghai 200090 Peoples R China

multimodal analysis of placental ultrasound (US) and microflow imaging (MFI) could greatly aid in the early diagnosis and interventional treatment of placental insufficiency (PI), ensuring a normal pregnancy. Existing multimodal analysis methods have weaknesses in multimodal feature representation and modal knowledge definitions and fail on incomplete datasets with unpaired multimodal samples. To address these challenges and efficiently leverage the incomplete multimodal dataset for accurate PI diagnosis, we propose a novel graph-based manifold regularization learning (MRL) framework named GMRLNet. It takes US and MFI images as input and exploits their modality-shared and modality-specific information for optimal multimodal feature representation. Specifically, a graph convolutional-based shared and specific transfer network (GSSTN) is designed to explore intra-modal feature associations, thus decoupling each modal input into interpretable shared and specific spaces. For unimodal knowledge definitions, graph-based manifold knowledge is introduced to describe the sample-level feature representation, local inter-sample relations, and global data distribution of each modality. Then, an MRL paradigm is designed for inter-modal manifold knowledge transfer to obtain effective cross-modal feature representations. Furthermore, MRL transfers the knowledge between both paired and unpaired data for robust learning on incomplete datasets. Experiments were conducted on two clinical datasets to validate the PI classification performance and generalization of GMRLNet. State-of-the-art comparisons show the higher accuracy of GMRLNet on incomplete datasets. Our method achieves 0.913 AUC and 0.904 balanced accuracy (bACC) for paired US and MFI images, as well as 0.906 AUC and 0.888 bACC for unimodal US images, illustrating its application potential in PI CAD systems.

关键词： incomplete multimodal learning graph neural network knowledge transfer prenatal diagnosis

来源：评论

学校读者我要写书评

暂无评论

Multiplex graph aggregation and feature refinement for unsupervised incomplete multimodal emotion recognition

引用

INFORMATION FUSION 2025年 114卷

作者： Deng, Yuanyue Bian, Jintang Wu, Shisong Lai, Jianhuang Xie, Xiaohua Sun Yat Sen Univ Sch Comp Sci & Engn Guangzhou Peoples R China Guangdong Prov Key Lab Informat Secur Technol Guangzhou Peoples R China Pazhou Lab Huangpu Guanghzou Peoples R China CSG AI Technol Coz Ltd Shenzhen Peoples R China

multimodal Emotion Recognition (MER) involves integrating information of various modalities, including audio, visual, text and physiological signals, to comprehensively grasp human sentiments, which has emerged as a vibrant area within human-computer interaction. Researchers have developed many methods for this but many of these methods rely on labeled supervised learning and struggle to address the issue of missing modalities of data. To address these issues, we propose a Multiplex Graph Aggregation and Feature Refinement framework for unsupervised incomplete MER, comprising four modules: Completion, Aggregation, Refinement, and Embedding. Specifically, we first capture the correlation information between samples using the structures, which aids in the completion of missing data and the multiplex aggregation of multimodal Then, we perform refinement operations on the aggregated features as well as alignment and enhancement operations on the embedding features to obtain the fused feature representations, which are consistent, highly separable and conducive to emotion recognition. Experimental results on multimodal emotion recognition datasets demonstrate that our method achieves state-of-the-art performance among unsupervised methods, validating its effectiveness.

关键词： incomplete multimodal learning Emotion recognition Multiplex graph aggregation Contrastive learning

来源：评论

学校读者我要写书评

暂无评论

Correlation-Decoupled Knowledge Distillation for multimodal Sentiment Analysis with incomplete Modalities

Correlation-Decoupled Knowledge Distillation for Multimodal ...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Li, Mingcheng Yang, Dingkang Zhao, Xiao Wang, Shuaibing Wang, Yan Yang, Kun Sun, Mingyang Kou, Dongliang Qian, Ziyun Zhang, Lihua Fudan Univ Acad Engn & Technol Shanghai Peoples R China Cognit & Intelligent Technol Lab CIT Lab Beijing Peoples R China Jilin Prov Key Lab Intelligence Sci & Engn Changchun Peoples R China Minist Educ Engn Res Ctr AI & Robot Shanghai Peoples R China

ISBN: (纸本)9798350353006

multimodal sentiment analysis (MSA) aims to understand human sentiment through multimodal data. Most MSA efforts are based on the assumption of modality completeness. However, in real-world applications, some practical factors cause uncertain modality missingness, which drastically degrades the model's performance. To this end, we propose a Correlation-decoupled Knowledge Distillation (CorrKD) framework for the MSA task under uncertain missing modalities. Specifically, we present a sample-level contrastive distillation mechanism that transfers comprehensive knowledge containing cross-sample correlations to reconstruct missing semantics. Moreover, a category-guided prototype distillation mechanism is introduced to capture cross-category correlations using category prototypes to align feature distributions and generate favorable joint representations. Eventually, we design a response-disentangled consistency distillation strategy to optimize the sentiment decision boundaries of the student network through response disentanglement and mutual information maximization. Comprehensive experiments on three datasets indicate that our framework can achieve favorable improvements compared with several baselines.

关键词： incomplete multimodal learning multimodal sentiment analysis

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：