检索结果-内蒙古大学图书馆

masked Autoencoders in Computer Vision: A Comprehensive Survey

IEEE ACCESS 2023年 11卷 113560-113579页

作者： Zhou, Zexian Liu, Xiaojing Qinghai Univ Dept Comp Technol & Applicat Xining 810016 Peoples R China

masked autoencoders (MAE) is a deep learning method based on Transformer. Originally used for images, it has now been extended to video, audio, and some other temporal prediction tasks. In the field of computer vision, MAE performs well in classification, prediction, and target detection tasks. In terms of specific application, MAE has made many achievements in medical treatment, geography, 3D point cloud and machine troubleshooting. Since its introduction at the end of 2021, there have been more than 300 related preprints, and MAE has been significantly performed in tier one computer vision conferences during 2022 and 2023. In view of the current popularity of MAE and its future development prospects, we conduct a relatively comprehensive survey of MAE mainly covering officially published articles so far. We comb through and classify the improvements in MAE, demonstrating relatively representative applications in computer vision. Finally, as a summary, we discuss the possible future research directions and development areas based on the characteristics of MAE, hoping our work could be a reference for the future work of MAE.

关键词： Computer vision survey MAE masked autoencoders masked image modeling

来源：评论

学校读者我要写书评

暂无评论

Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network With Token Migration

引用

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024年第12期46卷 9766-9779页

作者： Tian, Yunjie Xie, Lingxi Qiu, Jihao Jiao, Jianbin Wang, Yaowei Tian, Qi Ye, Qixiang Univ Chinese Acad Sci Sch Elect Elect & Commun Engn Beijing 101408 Peoples R China Huawei Inc Shenzhen 518129 Peoples R China Peng Cheng Lab Shenzhen 100013 Peoples R China

We propose integrally pre-trained transformer pyramid network (iTPN), towards jointly optimizing the network backbone and the neck, so that transfer gap between representation models and downstream tasks is minimal. iTPN is born with two elaborated designs: 1) The first pre-trained feature pyramid upon vision transformer (ViT). 2) Multi-stage supervision to the feature pyramid using masked feature modeling (MFM). iTPN is updated to Fast-iTPN, reducing computational memory overhead and accelerating inference through two flexible designs. 1) Token migration: dropping redundant tokens of the backbone while replenishing them in the feature pyramid without attention operations. 2) Token gathering: reducing computation cost caused by global attention by introducing few gathering tokens. The base/large-level Fast-iTPN achieve 88.75%/89.5% top-1 accuracy on imageNet-1 K. With 1x training schedule using DINO, the base/large-level Fast-iTPN achieves 58.4%/58.8% box AP on COCO object detection, and a 57.5%/58.7% mIoU on ADE20 K semantic segmentation using MaskDINO. Fast-iTPN can accelerate the inference procedure by up to 70%, with negligible performance loss, demonstrating the potential to be a powerful backbone for downstream vision tasks.

关键词： Downstream generalization integrally pre-trained masked image modeling transformer pyramid networks

来源：评论

学校读者我要写书评

暂无评论

AMSC: Adaptive Masking and Structure-Constraint Learning for Domain Adaptive Semantic Segmentation Under Adverse Conditions

引用

IEEE SIGNAL PROCESSING LETTERS 2024年 31卷 181-185页

作者： Chen, Yang Shan, Liang Qu, Yi Zhang, Weilong Chang, Lu Nanjing Univ Sci & Technol Sch Automat Nanjing 210094 Peoples R China

Unsupervised domain adaptation (UDA) aims to enable autonomous vehicles to understand complex road information without semantic labels. However, traditional UDA methods face challenges under adverse conditions, leading to suboptimal segmentation. These methods grapple with two core problems: distinguishing visually similar classes and misclassifying sudden dynamic classes in the target domain. To tackle these issues, we introduce Adaptive Masking and Structure-Constraint Learning (AMSC), comprising: 1) Adaptive Masking module for context leverage. 2) Structure-Constraint Learning to address structural loss from masking. Through these modules, we effectively tackle the aforementioned challenges. Simulation results showcase AMSC's state-of-the-art UDA performance, achieving a mIoU of 73.2 on Cityscapes to ACDC, a standard benchmark for normal-to-adverse semantic segmentation in UDA.

关键词： Semantic segmentation unsupervised learning domain adaptation deep learning masked image modeling

来源：评论

学校读者我要写书评

暂无评论

Context Autoencoder for Self-supervised Representation Learning

引用

INTERNATIONAL JOURNAL OF COMPUTER VISION 2023年第1期132卷 208-223页

作者： Chen, Xiaokang Ding, Mingyu Wang, Xiaodi Xin, Ying Mo, Shentong Wang, Yunhao Han, Shumin Luo, Ping Zeng, Gang Wang, Jingdong Peking Univ Beijing Peoples R China Univ Hong Kong Pok Fu Lam Hong Kong Peoples R China Univ Calif Berkeley Berkeley CA USA Baidu Beijing Peoples R China

We present a novel masked image modeling (MIM) approach, context autoencoder (CAE), for self-supervised representation pretraining. We pretrain an encoder by making predictions in the encoded representation space. The pretraining tasks include two tasks: masked representation prediction-predict the representations for the masked patches, and masked patch reconstruction-reconstruct the masked patches. The network is an encoder-regressor-decoder architecture: the encoder takes the visible patches as input;the regressor predicts the representations of the masked patches, which are expected to be aligned with the representations computed from the encoder, using the representations of visible patches and the positions of visible and masked patches;the decoder reconstructs the masked patches from the predicted encoded representations. The CAE design encourages the separation of learning the encoder (representation) from completing the pertaining tasks: masked representation prediction and masked patch reconstruction tasks, and making predictions in the encoded representation space empirically shows the benefit to representation learning. We demonstrate the effectiveness of our CAE through superior transfer performance in downstream tasks: semantic segmentation, object detection and instance segmentation, and classification. The code will be available at https://***/ Atten4Vis/CAE.

关键词： Self-supervised representation learning masked image modeling Context autoencoder

来源：评论

学校读者我要写书评

暂无评论

MFAE: masked Frequency Autoencoders for Domain Generalization Face Anti-Spoofing

引用

IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY 2024年 19卷 4058-4069页

作者： Zheng, Tianyi Li, Bo Wu, Shuang Wan, Ben Mu, Guodong Liu, Shice Ding, Shouhong Wang, Jia Shanghai Jiao Tong Univ Sch Elect Informat & Elect Engn Dept Elect Engn Shanghai 200240 Peoples R China Tencent YouTu Lab Shanghai 200233 Peoples R China

The generalizable face anti-spoofing (FAS) has attracted much attention recently. Even though many existing methods perform well under intra-domain settings, the model's performance in the unseen domain is not satisfying. In this paper, we shift our attention to the frequency domain to seek a solution. Specifically, we examine the characteristics of different frequency band components of FAS images and observe that the model's cross-domain performance is very sensitive to low-frequency features. To alleviate this sensitivity and improve the model's performance in FAS cross-domain tasks, we propose a new approach called masked Frequency Autoencoders (MFAE). MFAE randomly masks a portion of frequencies on the low-frequency spectrum of the image and then reconstructs the image from the resulting embedding. This innovative masked image modeling (MIM) strategy can be used as a self-supervised task for pre-training vision transformers (ViTs), which can reduce the ViT encoder's sensitivity to domain shifts. Additionally, we add an auxiliary content-regularization decoder in our MFAE to encourage the encoder to be insensitive to low-frequency features. The results show that the model insensitive to low-frequency features performs well on extensive public datasets and outperforms other state-of-the-art methods in cross-domain FAS tasks.

关键词： Face anti-spoofing masked image modeling self-supervised frequency domain

来源：评论

学校读者我要写书评

暂无评论

AST: Adaptive Self-supervised Transformer for optical remote sensing representation

引用

ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING 2023年第1期200卷 41-54页

作者： He, Qibin Sun, Xian Yan, Zhiyuan Wang, Bing Zhu, Zicong Diao, Wenhui Yang, Michael Ying Chinese Acad Sci Aerosp Informat Res Inst Beijing 100190 Peoples R China Chinese Acad Sci Aerosp Informat Res Inst Key Lab Network Informat Syst Technol NIST Beijing 100190 Peoples R China Univ Chinese Acad Sci Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Elect Elect & Commun Engn Beijing 100190 Peoples R China Univ Twente Fac Geoinformat Sci & Earth Observ ITC Enschede Netherlands

Due to the variation in spatial resolution and the diversity of object scales, the interpretation of optical remote sensing images is extremely challenging. Deep learning has become the mainstream solution to interpret such complex scenes. However, the explosion of deep learning model architectures has resulted in the need for hundreds of millions of remote sensing images for which labels are very costly or often unavailable publicly. This paper provides an in-depth analysis of the main reasons for this data thirst, i.e., (i) limited representational power for model learning, and (ii) underutilization of unlabeled remote sensing data. To overcome the above difficulties, we present a scalable and adaptive self-supervised Transformer (AST) for optical remote sensing image interpretation. By performing masked image modeling in pre-training, the proposed AST releases the rich supervision signals in massive unlabeled remote sensing data and learns useful multi-scale semantics. Specifically, a cross-scale Transformer architecture is designed to collaboratively learn global dependencies and local details by introducing a pyramid structure, to facilitate multi-granular feature interactions and generate scale-invariant representations. Furthermore, a masking token strategy relying on correlation mapping is proposed to achieve adaptive masking of partial patches without affecting key structures, which enhances the understanding of visually important regions. Extensive experiments on various optical remote sensing interpretation tasks show that AST has good generalization capability and competitiveness.

关键词： Cross-scale transformer Interpretation masked image modeling Optical remote sensing Representation learning

来源：评论

学校读者我要写书评

暂无评论

Learning Features of Intra-Consistency and Inter-Diversity: Keys Toward Generalizable Deepfake Detection

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2023年第3期33卷 1468-1480页

作者： Chen, Han Lin, Yuzhen Li, Bin Tan, Shunquan Shenzhen Univ Guangdong Key Lab Intelligent Informat Proc Shenzhen Key Lab Media Secur Shenzhen 518060 Peoples R China Shenzhen Univ Guangdong Lab Artificial Intelligence & Digital Ec Shenzhen 518060 Peoples R China

Public concerns about deepfake face forgery are continually rising in recent years. Most deepfake detection approaches attempt to learn discriminative features between real and fake faces through end-to-end trained deep neural networks. However, the majorities of them suffer from the problem of poor generalization across different data sources, forgery methods, and/or post-processing operations. In this paper, following the simple but effective principle in discriminative representation learning, i.e., towards learning features of intra-consistency within classes and inter-diversity between classes, we leverage a novel transformer-based self-supervised learning method and an effective data augmentation strategy towards generalizable deepfake detection. Considering the differences between the real and fake images are often subtle and local, the proposed method firstly utilizes Self Prediction Learning (SPL) to learn rich hidden representations by predicting masked patches at a pre-training stage. Intra-class consistency clues in images can be mined without deepfake labels. After pre-training, the discrimination model is then fine-tuned via multi-task learning, including a deepfake classification task and a forgery mask estimation task. It is facilitated by our new data augmentation method called Adjustable Forgery Synthesizer (AFS), which can conveniently simulate the process of synthesizing deepfake images with various levels of visual reality in an explicit manner. AFS greatly prevents overfitting due to insufficient diversity in training data. Comprehensive experiments demonstrate that our method outperforms the state-of-the-art competitors on several popular benchmark datasets in terms of generalization to unseen forgery methods and untrained datasets.

关键词： Deepfakes Task analysis Forgery Faces Feature extraction Data models Transformers Deepfake detection self-supervised learning masked image modeling transformer generalization

来源：评论

学校读者我要写书评

暂无评论

AWDepth: Monocular Depth Estimation for Adverse Weather via masked Encoding

引用

IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS 2024年第9期20卷 10873-10882页

作者： Wang, Meng Qin, Yunchuan Li, Ruihui Liu, Zhizhong Tang, Zhuo Li, Kenli Hunan Univ Coll Comp Sci & Elect Engn Changsha 410082 Peoples R China

Monocular depth estimation has made considerable advances under clear weather conditions. However, how to learn accurate scene depth under rain and fog conditions and alleviate the negative influence of occlusion, light, visibility, etc., is an open problem. To address this problem, in this article, we split the adverse weather depth estimation network into two subbranches: the depth prediction branch and the masked encoding branch. The depth prediction branch is used for depth estimation. The masked encoding branch, inspired by masked image modeling, uses random masks to simulate occlusion or low visibility often seen in rain and fog, forcing this branch to learn to infer the prediction of masked regions from the context. In order to make the masked encoding better enhance the depth prediction, we designed the mask feature fusion module, which can fuse the depth and spatial context features of the two branches to produce a fine-level depth map. The experimental results on the Foggy Cityscapes and RainCityscapes datasets demonstrate that our method achieves state-of-the-art performance, significantly outperforming previous methods across all evaluation metrics.

关键词： Estimation Meteorology Task analysis Feature extraction Transformers image coding image reconstruction Depth estimation masked image modeling swin transformer

来源：评论

学校读者我要写书评

暂无评论

Generic-to-Specific Distillation of masked Autoencoders

引用

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 2024年第9期34卷 8779-8793页

作者： Huang, Wei Peng, Zhiliang Dong, Li Wei, Furu Ye, Qixiang Jiao, Jianbin Univ Chinese Acad Sci Sch Elect Elect & Commun Engn Beijing 100049 Peoples R China Microsoft Res Beijing 100080 Peoples R China

To transfer the representation capacity of large pre-trained models to lightweight models, knowledge distillation has been widely explored. However, conventional single-stage distillation methods are prone to getting stuck in the transfer of task-specific knowledge, making it difficult to retain task-agnostic knowledge which is crucial for model generalization. In this study, we propose generic-to-specific distillation (G2SD), to boost lightweight models under the assistance of large models pre-trained by masked image modeling. In generic distillation, the decoder of a small model is encouraged to align feature predictions with that of a large model, so that task-agnostic knowledge can be transferred. In specific distillation, predictions of the small model are encouraged to be consistent with those of the large model, to guarantee task performance. G2SD is also applicable for heterogeneous settings(i.e., distilling from ViT to CNN). With G2SD, the ViT-Small model respectively achieves 98.9%, 98.4%, 99.3% and 98.9% accuracies when compared with its teachers (ViT-Base) for image classification, object detection, semantic segmentation and video recognition tasks. The lightweight ResNet models are improved to a new height on image classification task. The code is available at ***/pengzhiliang/G2SD.

关键词： Knowledge distillation masked image modeling heterogeneous-architecture distillation

来源：评论

学校读者我要写书评

暂无评论

Transferable adversarial masked self-distillation for unsupervised domain adaptation

引用

COMPLEX & INTELLIGENT SYSTEMS 2023年第6期9卷 6567-6580页

作者： Xia, Yuelong Yun, Li-Jun Yang, Chengfu Yunnan Normal Univ Sch Informat Sci & Technol Kunming 650500 Peoples R China Engn Res Ctr Comp Vis & Intelligent Control Techno Dept Educ Yunnan Prov Kunming 650500 Peoples R China

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to a related unlabeled target domain. Most existing works focus on minimizing the domain discrepancy to learn global domain-invariant representation using CNN-based architecture while ignoring both transferable and discriminative local representation, e.g, pixel-level and patch-level representation. In this paper, we propose the Transferable Adversarial masked Self-distillation based on Vision Transformer architecture to enhance the transferability of UDA, named TAMS. Specifically, TAMS jointly optimizes three objectives to learn both task-specific class-level global representation and domain-specific local representation. First, we introduce adversarial masked self-distillation objective to distill representation from a full image to the representation predicted from a masked image, which aims to learn task-specific global class-level representation. Second, we introduce masked image modeling objectives to learn local pixel-level representation. Third, we introduce an adversarial weighted cross-domain adaptation objective to capture discriminative potentials of patch tokens, which aims to learn both transferable and discriminative domain-specific patch-level representation. Extensive studies on four benchmarks and the experimental results show that our proposed method can achieve remarkable improvements compared to previous state-of-the-art UDA methods.

关键词： Unsupervised domain adaptation masked self-distillation masked image modeling Adversarial weighted cross-domain adaptation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：