检索结果-内蒙古大学图书馆

Hybrid Graph Convolutional Network With Online masked autoencoder for Robust Multimodal Cancer Survival Prediction

IEEE TRANSACTIONS ON MEDICAL IMAGING 2023年第8期42卷 2462-2473页

作者： Hou, Wentai Lin, Chengxuan Yu, Lequan Qin, Jing Yu, Rongshan Wang, Liansheng Xiamen Univ Sch Informat Dept Informat & Commun Engn Xiamen 361005 Peoples R China Xiamen Univ Natl Inst Data Sci Hlth & Med Xiamen 361102 Peoples R China Xiamen Univ Sch Informat Dept Comp Sci Xiamen 361005 Peoples R China Univ Hong Kong Dept Stat & Actuarial Sci Hong Kong Peoples R China Hong Kong Polytech Univ Ctr Smart Hlth Sch Nursing Hong Kong Peoples R China

Cancer survival prediction requires exploiting related multimodal information (e.g., pathological, clinical and genomic features, etc.) and it is even more challenging in clinical practices due to the incompleteness of patient's multimodal data. Furthermore, existing methods lack sufficient intra- and inter-modal interactions, and suffer from significant performance degradation caused by missing modalities. This manuscript proposes a novel hybrid graph convolutional network, entitled HGCN, which is equipped with an online masked autoencoder paradigm for robust multimodal cancer survival prediction. Particularly, we pioneer modeling the patient's multimodal data into flexible and interpretable multimodal graphs with modality-specific preprocessing. HGCN integrates the advantages of graph convolutional networks (GCNs) and a hypergraph convolutional network (HCN) through node message passing and a hyperedge mixing mechanism to facilitate intra-modal and inter-modal interactions between multimodal graphs. With HGCN, the potential for multimodal data to create more reliable predictions of patient's survival risk is dramatically increased compared to prior methods. Most importantly, to compensate for missing patient modalities in clinical scenarios, we incorporated an online masked autoencoder paradigm into HGCN, which can effectively capture intrinsic dependence between modalities and seamlessly generate missing hyperedges for model inference. Extensive experiments and analysis on six cancer cohorts from TCGA show that our method significantly outperforms the state-of-the-arts in both complete and missing modal settings. Our codes are made available at https://***/lin-lcx/HGCN.

关键词： Survival prediction multi-modal learning graph convolutional network hypergraph convolutional network masked autoencoder decision fusion

来源：评论

学校读者我要写书评

暂无评论

Self-Supervised Learning Malware Traffic Classification Based on masked autoencoder

引用

IEEE INTERNET OF THINGS JOURNAL 2024年第10期11卷 17330-17340页

作者： Xu, Ke Zhang, Xixi Wang, Yu Ohtsuki, Tomoaki Adebisi, Bamidele Sari, Hikmet Gui, Guan Nanjing Univ Posts & Telecommun Coll Telecommun & Informat Engn Nanjing 210003 Peoples R China Keio Univ Dept Informat & Comp Sci Minato 1080073 Japan Manchester Metropolitan Univ Fac Sci & Engn Dept Engn Manchester M1 5GD England

Malware traffic classification (MTC) is one of the important techniques to ensure the security of cyberspace, which aims to detect anomalies and classify different types of network traffic. Recently, MTC methods based on deep learning (DL) have shown their excellent performance. However, these DL-based methods rely on data sets with manually labeled samples for training, which are costly and hard to obtain. To address this problem, this article proposes a novel self-supervised MTC method based on the framework of masked auto-encoder (MAE). Specifically, MAE first constructs a reasonable unsupervised pretext task with a random masking strategy, which reduces the redundant information in samples and speeds up the pretraining process. The transformer-based backbone network then efficiently extracts features from the nonredundant traffic data efficiently. The proposed MTC-MAE method employs self-supervised learning on a large-scale unlabeled data set to acquire unbiased features, and fine-tunes on specific data sets to adapt to diverse traffic classification scenarios. Simulation experiments show that our proposed MTC-MAE method is able to learn universal features with high quality and has excellent classification performance on various downstream data sets. The data sets we used, code implementation, and pretrained models are available on GitHub. Code available at https://***/TsuiHark/Self-supervised_MTC.

关键词： Task analysis Feature extraction Malware Internet of Things Training Self-supervised learning Cryptography Malware traffic classification (MTC) masked autoencoder self-supervised learning (SSL) transformer

来源：评论

学校读者我要写书评

暂无评论

MASA: Motion-Aware masked autoencoder With Semantic Alignment for Sign Language Recognition

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2024年第11期34卷 10793-10804页

作者： Zhao, Weichao Hu, Hezhen Zhou, Wengang Mao, Yunyao Wang, Min Li, Houqiang Univ Sci & Technol China Dept Elect Engn & Informat Sci CAS Key Lab Technol Geospatial Informat Proc & Ap Hefei 230027 Peoples R China Univ Texas Austin Visual Informat Grp Austin TX 78705 USA Hefei Comprehens Natl Sci Ctr Inst Artificial Intelligence Hefei 230030 Peoples R China

Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performance by employing various pretext tasks on sign pose data, these methods still suffer from two primary limitations: i) Explicit motion information is usually disregarded in previous pretext tasks, leading to partial information loss and limited representation capability. ii) Previous methods focus on the local context of a sign pose sequence, without incorporating the guidance of the global meaning of lexical signs. To this end, we propose a MotionAware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information in a self-supervised learning paradigm for SLR. Our framework contains two crucial components, i.e., a motion-aware masked autoencoder (MA) and a momentum semantic alignment module (SA). Specifically, in MA, we introduce an autoencoder architecture with a motion-aware masked strategy to reconstruct motion residuals of masked frames, thereby explicitly exploring dynamic motion cues among sign pose sequences. Moreover, in SA, we embed our framework with global semantic awareness by aligning the embeddings of different augmented samples from the input sequence in the shared latent space. In this way, our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation. Furthermore, we conduct extensive experiments to validate the effectiveness of our method, achieving new stateof-the-art performance on four public benchmarks. The source code are publicly available at https://***/sakura/MASA.

关键词： masked autoencoder motion-aware semantic alignment sign language recognition.

来源：评论

学校读者我要写书评

暂无评论

GAF-MAE: A Self-Supervised Automatic Modulation Classification Method Based on Gramian Angular Field and masked autoencoder

引用

IEEE TRANSACTIONS ON COGNITIVE COMMUNICATIONS AND NETWORKING 2024年第1期10卷 94-106页

作者： Shi, Yunhao Xu, Hua Zhang, Yue Qi, Zisen Wang, Dan Air Force Engn Univ Xian 710038 Peoples R China Northwestern Polytech Univ Unmanned Syst Res Inst Xian 710072 Peoples R China

With the development of deep learning (DL), several fields have ushered in leapfrog development, such as image classification and natural language processing. Combing the powerful feature extraction tool, DL-based Automatic Modulation Classification (AMC) emerges. However, most DL-based AMC methods require massive labeled samples for training, which is difficult for non-cooperative scenarios. In this paper, we propose a novel self-supervised AMC method called GAF-MAE. Gramian Angular Field (GAF) is applied for domain transformation from time series to images first, then a masked autoencoder (MAE) is built for self-supervised reconstruction tasks through unlabeled samples. After self-supervised pretrain, a small number of labeled samples are used for downstream classification finetuning. We conduct a comprehensive evaluation of a public dataset RadioML.2016.10a. The simulation results show that GAF-MAE can achieve a relatively high average accuracy of 54.85% even if the label proportion is 5% and 61.38% when the label proportion reaches 100%, which outperforms other well-known models.

关键词： Automatic modulation classification masked autoencoder self-supervised

来源：评论

学校读者我要写书评

暂无评论

GMAEEG: A Self-Supervised Graph masked autoencoder for EEG Representation Learning

引用

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 2024年第11期28卷 6486-6497页

作者： Fu, Zanhao Zhu, Huaiyu Zhao, Yisheng Huan, Ruohong Zhang, Yi Chen, Shuohui Pan, Yun Zhejiang Univ Coll Informat Sci & Elect Engn Hangzhou 310027 Peoples R China Zhejiang Univ Technol Coll Comp Sci & Technol Hangzhou 310023 Peoples R China Zhejiang Univ Childrens Hosp Sch Med Dept Neurol Hangzhou 310052 Peoples R China Zhejiang Univ Childrens Hosp Sch Med Nursing Dept Hangzhou 310052 Peoples R China

Annotated electroencephalogram (EEG) data is the prerequisite for artificial intelligence-driven EEG autoanalysis. However, the scarcity of annotated data due to its high-cost and the resulted insufficient training limits the development of EEG autoanalysis. Generative self-supervised learning, represented by masked autoencoder, offers potential but struggles with non-Euclidean structures. To alleviate these challenges, this work proposes a self-supervised graph masked autoencoder for EEG representation learning, named GMAEEG. Concretely, a pretrained model is enriched with temporal and spatial representations through a masked signal reconstruction pretext task. A learnable dynamic adjacency matrix, initialized with prior knowledge, adapts to brain characteristics. Downstream tasks are achieved by finetuning pretrained parameters, with the adjacency matrix transferred based on task functional similarity. Experimental results demonstrate that with emotion recognition as the pretext task, GMAEEG reaches superior performance on various downstream tasks, including emotion, major depressive disorder, Parkinson's disease, and pain recognition. This study is the first to tailor the masked autoencoder specifically for EEG representation learning considering its non-Euclidean characteristics. Further, graph connection analysis based on GMAEEG may provide insights for future clinical studies.

关键词： Electroencephalography Task analysis Representation learning Self-supervised learning Emotion recognition Convolution Image reconstruction EEG graph network masked autoencoder representation learning self-supervised learning

来源：评论

学校读者我要写书评

暂无评论

Rethinking Vision Transformer and masked autoencoder in Multimodal Face Anti-Spoofing

引用

INTERNATIONAL JOURNAL OF COMPUTER VISION 2024年第11期132卷 5217-5238页

作者： Yu, Zitong Cai, Rizhao Cui, Yawen Liu, Xin Hu, Yongjian Kot, Alex C. Great Bay Univ Sch Comp & Informat Technol Dongguan 523000 Peoples R China Shenzhen Univ Natl Engn Lab Big Data Syst Comp Technol Shenzhen 518060 Peoples R China Nanyang Technol Univ ROSE Lab Singapore Singapore Lappeenranta Lahti Univ Technol Lappeenranta 53850 Finland South China Univ Technol Guangzhou 510000 Peoples R China

Recently, vision transformer (ViT) based multimodal learning methods have been proposed to improve the robustness of face anti-spoofing (FAS) systems. However, there are still no works to explore the fundamental natures (e.g., modality-aware inputs, suitable multimodal pre-training, and efficient finetuning) in vanilla ViT for multimodal FAS. In this paper, we investigate three key factors (i.e., inputs, pre-training, and finetuning) in ViT for multimodal FAS with RGB, Infrared (IR), and Depth. First, in terms of the ViT inputs, we find that leveraging local feature descriptors (such as histograms of oriented gradients) benefits the ViT on IR modality but not RGB or Depth modalities. Second, in consideration of the task (FAS vs. generic object classification) and modality (multimodal vs. unimodal) gaps, ImageNet pre-trained models might be sub-optimal for the multimodal FAS task. Finally, in observation of the inefficiency on direct finetuning the whole or partial ViT, we design an adaptive multimodal adapter (AMA), which can efficiently aggregate local multimodal features while freezing majority of ViT parameters. To bridge these gaps, we propose the modality-asymmetric masked autoencoder (M 2 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$<^>{2}$$\end{document} A 2 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$<^>{2}$$\end{document} E) for multimodal FAS self-supervised pre-training without costly annotated labels. Compared with the previous modality-symmetric autoencoder, the proposed M 2 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \

关键词： Multimodal Face anti-spoofing Adaptive multimodal adapter masked autoencoder

来源：评论

学校读者我要写书评

暂无评论

HSIMAE: A Unified masked autoencoder With Large-Scale Pretraining for Hyperspectral Image Classification

引用

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING 2024年 17卷 14064-14079页

作者： Wang, Yue Wen, Ming Zhang, Hailiang Sun, Jinyu Yang, Qiong Zhang, Zhimin Lu, Hongmei Cent South Univ Coll Chem & Chem Engn Changsha 410083 Peoples R China

With a spurt of progress in deep learning techniques, convolutional neural network-based and transformer-based methods have yielded impressive performance on the hyperspectral image (HSI) classification tasks. However, pixel-level manual annotation is time-consuming and laborious, and the small amount of labeled HSI data brings challenges to deep learning methods. Existing methods use carefully designed network architectures combined with self-supervised or semi-supervised learning to deal with the lack of training samples. Those methods were designed for specific datasets and often needed to tune hyperparameters on new datasets carefully. To tackle this problem, a unified HSI masked autoencoder framework was proposed for HSI classification. Different from existing works, the hyperspectral image masked autoencoder (HSIMAE) framework was pretrained on a large-scale unlabeled HSI dataset, named HSIHybrid, which contained a large amount of HSI data acquired by different sensors. First, to handle the different spectral ranges of HSIs, a group-wise PCA was applied to extract features of HSI spectra and transform them into fixed-length vectors. Then, a modified masked autoencoder was proposed for large-scale pretraining. It utilized separate spatial-spectral encoders followed by fusion blocks to learn spatial correlation and spectral correlation of HSI data. Finally, to leverage the unlabeled data of the target dataset, a dual-branch finetuning framework that used an extra unlabeled branch for mask modeling learning was introduced. Extensive experiments were conducted on four HSI datasets from different hyperspectral sensors. The results demonstrate the superiority of the proposed HSIMAE framework over the state-of-the-art methods, even with very few training samples.

关键词： Hyperspectral image (HSI) classification large-scale pretraining masked autoencoder self-supervised learning transformer Hyperspectral image (HSI) classification large-scale pretraining masked autoencoder self-supervised learning transformer

来源：评论

学校读者我要写书评

暂无评论

HiCMAE: Hierarchical Contrastive masked autoencoder for self-supervised Audio-Visual Emotion Recognition

引用

INFORMATION FUSION 2024年 108卷

作者： Sun, Licai Lian, Zheng Liu, Bin Tao, Jianhua Univ Chinese Acad Sci Sch Artificial Intelligence Beijing Peoples R China Chinese Acad Sci Inst Automat Beijing Peoples R China Tsinghua Univ Dept Automat Beijing Peoples R China Tsinghua Univ Beijing Natl Res Ctr Informat Sci & Technol Beijing Peoples R China

Audio -Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion -aware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self -supervised learning, we propose Hierarchical Contrastive masked autoencoder (HiCMAE), a novel self -supervised framework that leverages large-scale self -supervised pre -training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self -supervised audio-visual representation learning, HiCMAE adopts two primary forms of self -supervision for pre -training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top -layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three -pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. Firstly, it incorporates hierarchical skip connections between the encoder and decoder to encourage intermediate layers to learn more meaningful representations and bolster masked audio-visual reconstruction. Secondly, hierarchical cross -modal contrastive learning is also exerted on intermediate representations to narrow the audio-visual modality gap progressively and facilitate subsequent cross -modal fusion. Finally, during downstream fine-tuning, HiCMAE employs hierarchical feature fusion to comprehensively integrate multi -level features from different layers. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self -supervised audio-visual methods, whi

关键词： Audio-Visual Emotion Recognition Self-supervised learning masked autoencoder Contrastive learning

来源：评论

学校读者我要写书评

暂无评论

Inter-Modal masked autoencoder for Self-Supervised Learning on Point Clouds

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 3897-3908页

作者： Liu, Jiaming Wu, Yue Gong, Maoguo Liu, Zhixiao Miao, Qiguang Ma, Wenping Xidian Univ Sch Comp Sci & Technol Key Lab Collaborat Intelligence Syst Minist Educ Xian 710071 Peoples R China Xidian Univ Sch Elect Engn Key Lab Collaborat Intelligence Syst Minist Educ Xian 710071 Peoples R China Harbin Engn Univ Yantai Res Inst Yantai 264006 Peoples R China Xidian Univ Sch Artificial Intelligence Key Lab Intelligent Percept & Image Understanding Minist Educ Xian 710071 Peoples R China

masked autoencoder (MAE) is a recently widely used self-supervised learning method that has achieved great success in NLP and computer vision. However, the potential advantages of masked pre-training for point cloud understanding have not been fully explored. There is preliminary work on MAE-based point clouds using the Transformer architecture to explore low-level geometric representations in 3D space, which is insufficient for fine-grained decoding completion and downstream tasks. Inspired by multimodality, we propose Inter-MAE, a inter-modal MAE method for self-supervised learning on point clouds. Specifically, we first use Point-MAE as a baseline to partition point clouds into random low percentage of visible and high percentage of masked point patches. Then, a standard Transformer-based autoencoder is built by asymmetric design and shifting mask operations, and latent features are learned from the visible point patches aiming to recover the masked point patches. In addition, we generate image features based on ViT after point cloud rendering to form inter-modal contrastive learning with the decoded features of the completed point patches. Extensive experiments show that the proposed Inter-MAE generates pre-trained models that are effective and exhibit superior results in various downstream tasks. For example, an accuracy of 85.4% is achieved on ScanObjectNN and 86.3% on ShapeNetPart, outperforming other state-of-the-art self-supervised learning methods. Notably, our work establishes for the first time the feasibility of applying image modality to masked point clouds.

关键词： Point cloud compression Transformers Task analysis Standards Computer architecture Decoding Self-supervised learning Self-supervision masked autoencoder joint multimodality point cloud understanding

来源：评论

学校读者我要写书评

暂无评论

Audio Super-Resolution With Robust Speech Representation Learning of masked autoencoder

引用

IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING 2024年 32卷 1012-1022页

作者： Kim, Seung-Bin Lee, Sang-Hoon Choi, Ha-Yeong Lee, Seong-Whan Korea Univ Dept Artificial Intelligence Seoul 02841 South Korea

This paper proposes Fre-Painter, a high-fidelity audio super-resolution system that utilizes robust speech representation learning with various masking strategies. Recently, masked autoencoders have been found to be beneficial in learning robust representations of audio for speech classification tasks. Following these studies, we leverage these representations and investigate several masking strategies for neural audio super-resolution. In this paper, we propose an upper-band masking strategy with the initialization of the mask token, which is simple but efficient for audio super-resolution. Furthermore, we propose a mix-ratio masking strategy that makes the model robust for input speech with various sampling rates. For practical applicability, we extend Fre-Painter to a text-to-speech system, which synthesizes high-resolution speech using low-resolution speech data. The experimental results demonstrate that Fre-Painter outperforms other neural audio super-resolution models.

关键词： Superresolution Task analysis Speech processing Self-supervised learning Computational modeling Decoding Training Audio super-resolution bandwidth extension self-supervised learning masked autoencoder audio synthesis

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：