检索结果-内蒙古大学图书馆

An Experimental Study on Exploring Strong Lightweight Vision Transformers via masked image modeling Pre-training

INTERNATIONAL JOURNAL OF COMPUTER VISION 2025年 1-33页

作者： Gao, Jin Lin, Shubo Wang, Shaoru Kou, Yutong Li, Zeming Li, Liang Zhang, Congxuan Zhang, Xiaoqin Wang, Yizheng Hu, Weiming Chinese Acad Sci Inst Automat State Key Lab Multimodal Artificial Intelligence S Beijing 100190 Peoples R China Univ Chinese Acad Sci Sch Artificial Intelligence Beijing 101408 Peoples R China ShanghaiTech Univ Sch Informat Sci & Technol Shanghai 201210 Peoples R China Megvii Res Beijing 100089 Peoples R China Beijing Inst Basic Med Sci Beijing 100850 Peoples R China Nanchang Hangkong Univ Nanchang 330063 Peoples R China Zhejiang Univ Technol Hangzhou 310014 Peoples R China

masked image modeling (MIM) pre-training for large-scale vision transformers (ViTs) has enabled promising downstream performance on top of the learned self-supervised ViT features. In this paper, we question if the extremely simple lightweight ViTs' fine-tuning performance can also benefit from this pre-training paradigm, which is considerably less studied yet in contrast to the well-established lightweight architecture design methodology. We use an observation-analysis-solution flow for our study. We first systematically observe different behaviors among the evaluated pre-training methods with respect to the downstream fine-tuning data scales. Furthermore, we analyze the layer representation similarities and attention maps across the obtained models, which clearly show the inferior learning of MIM pre-training on higher layers, leading to unsatisfactory transfer performance on data-insufficient downstream tasks. This finding is naturally a guide to designing our distillation strategies during pre-training to solve the above deterioration problem. Extensive experiments have demonstrated the effectiveness of our approach. Our pre-training with distillation on pure lightweight ViTs with vanilla/hierarchical design (5.7M/6.5M) can achieve 79.4%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$79.4\%$$\end{document}/78.9%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$78.9\%$$\end{document} top-1 accuracy on imageNet-1K. It also enables SOTA performance on the ADE20K segmentation task (42.8%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage

关键词： masked image modeling Vision transformers Lightweight networks Knowledge distillation Visual tracking

来源：评论

学校读者我要写书评

暂无评论

MaDis-Stereo: Enhanced Stereo Matching via Distilled masked image modeling

引用

IEEE ACCESS 2025年 13卷 8912-8923页

作者： Ahn, Jihye Choi, Hyesong Kim, Soomin Min, Dongbo Ewha Womans Univ Dept Comp Sci & Engn Seoul 03760 South Korea

In stereo matching, Convolutional Neural Networks (CNNs), a class of deep learning models designed to process grid-like data such as images, have traditionally served as the predominant architectures. Although Transformer-based stereo models have been studied recently, their performance still lags behind CNN-based stereo models due to the inherent data scarcity issue in the stereo matching task. In this paper, we propose masked image modeling Distilled Stereo matching model, termed MaDis-Stereo, that enhances locality inductive bias by leveraging masked image modeling (MIM), a technique that improves model performance by randomly masking parts of the input image in training Transformer-based stereo model. Given randomly masked stereo images as inputs, our method attempts to conduct both image reconstruction and depth prediction tasks. While this strategy is beneficial to resolving the data scarcity issue, the dual tasks of reconstructing masked tokens and subsequently performing stereo matching pose significant challenges, particularly in terms of training stability. To address this, we propose to use an auxiliary network (teacher) that is updated via Exponential Moving Average (EMA), a method for updating model weights by averaging previous weights with a decaying factor. Specifically, the teacher knowledge is distilled and transferred to the original stereo model (student) by providing pseudo supervisory signals, enhancing training stability and overall model performance. Experimentally, our approach achieves state-of-the-art results on the ETH3D benchmark and competitive performance on the KITTI 2015 benchmark. Our findings highlight the potential for extending this approach to other vision tasks, such as object detection and semantic segmentation, that require sufficient locality inductive bias in Transformer-based architectures. Code is available at: https://***/ja053199/madis/

关键词： Transformers image reconstruction Training Depth measurement Head Data models Adaptation models Self-supervised learning Predictive models Training data Exponential moving average masked image modeling pseudo depth label stereo depth estimation

来源：评论

学校读者我要写书评

暂无评论

SRMAE: masked image modeling for Scale-Invariant Deep Representations 7th

SRMAE: Masked Image Modeling for Scale-Invariant Deep Repres...

引用

7th Chinese Conference on Pattern Recognition and Computer Vision

作者： Wang, Zhiming Gu, Lin Lu, Feng Beihang Univ Sch CSE State Key Lab VR Technol & Syst Beijing Peoples R China Univ Tokyo Tokyo Japan RIKEN AIP Chuo City Japan

ISBN: (纸本)9789819784899;9789819784905

Due to the prevalence of scale variance in nature images, we propose to use image scale as a self-supervised signal for masked image modeling (MIM). Our method involves selecting random patches from the input image and downsampling them to a low resolution format. Our framework utilizes the latest advances in super-resolution (SR) to design the prediction head, which reconstructs the input from low resolution clues and other patches. image scale signal also allows our SRMAE to capture scale-invariance feature. Experimental results demonstrate that utilizing scale variance as a self-supervised signal enhances the performance and robustness of SRMAE in various resolution-sensitive visual tasks compared with previous MIM methods. Moreover, our SRMAE outperforms the state-of-the-art supervised method DeriveNet by 1.3% in very low-resolution (VLR) recognition tasks and achieves an accuracy of 74.84% in the recognition of low-resolution facial expressions, surpassing the current leading supervised method FMD by 9.48%.

关键词： masked image modeling Super resolution VLR/LR recognition

来源：评论

学校读者我要写书评

暂无评论

CurMIM: Curriculum masked image modeling

CurMIM: Curriculum Masked Image Modeling

引用

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Liu, Hao Wang, Kun Han, Yudong Wang, Haocong Hu, Yupeng Wang, Chunxiao Nie, Liqiang School of Software Shandong University Jinan China Key Laboratory of Computing Power Network and Information Security Ministry of Education Qilu University of Technology Shandong Academy of Sciences Jinan China School of Computer Science and Technology Harbin Institute of Technology Shenzhen China

ISBN: (纸本)9798350368741

masked image modeling (MIM), following "mask-and-reconstruct" scheme, is a promising self-supervised method to learn scalable visual representation. Studies indicate that selecting an effective mask strategy is vital for MIM. However, existing approaches often rely on static pre-defined priors, which limit their ability to adapt mask strategies dynamically for network optimization. In this paper, we focus on the learning process of the network and introduce human-like curriculum into MIM for dynamic representation refinement, and propose an end-to-end framework Curriculum masked image modeling (CurMIM). CurMIM consists of two components: Mask Priority Measurer, which acts as a curriculum learner to determine mask priority values using the network's intrinsic state information, and Dual Adaptive Selector, which serves as a curriculum scheduler to create effective masks based on these values. With negligible extra parameters, our curriculum-based method consistently establishes noticeable improvements across varying model sizes and benchmarks, showing effectiveness and generalization. © 2025 IEEE.

关键词： Curriculum Learning image Representation masked image modeling Visual Pre-training

来源：评论

学校读者我要写书评

暂无评论

Toward accurate hand mesh estimation via masked image modeling

引用

FRONTIERS IN PHYSICS 2025年 12卷

作者： Li, Yanli Wang, Congyi Wang, Huan Nanchang Univ Fuzhou Med Coll Fuzhou Peoples R China Ind Bank Financial Technol Res Inst Fuzhou Peoples R China Guangdong Inst Sci & Tech Informat Ind Technol Res Ctr Guangzhou Peoples R China

Introduction With an enormous number of hand images generated over time, leveraging unlabeled images for pose estimation is an emerging yet challenging topic. While some semi-supervised and self-supervised methods have emerged, they are constrained by their reliance on high-quality keypoint detection models or complicated network *** We propose a novel selfsupervised pretraining strategy for 3D hand mesh regression. Our approach integrates a multi-granularity strategy with pseudo-keypoint alignment in a teacher-student framework, employing self-distillation and masked image modeling for comprehensive representation learning. We pair this with a robust pose estimation baseline, combining a standard vision transformer backbone with a pyramidal mesh alignment feedback *** Extensive experiments demonstrate HandMIM's competitive performance across diverse datasets, notably achieving an 8.00 mm Procrustes alignment vertex-point-error on the challenging HO3Dv2 test set, which features severe hand occlusions, surpassing many specially optimized architectures.

关键词： 3D hand mesh estimation multi-granularity representation self-supervised learning masked image modeling vision transformer

来源：评论

学校读者我要写书评

暂无评论

Optimizing agricultural classification with masked image modeling

引用

COGENT FOOD & AGRICULTURE 2025年第1期11卷

作者： Peng, Ying shu Wang, Yi Hunan Acad Agr Sci Hort Res Inst Changsha 410125 Peoples R China Yuelu Mt Lab Changsha Peoples R China Chinese Acad Sci Lushan Bot Garden Beijing Jiangxi Peoples R China Jiangsu Adv Energy Storage Technol Co Ltd Battery Technol Dept Nanjing Peoples R China

image classification poses a significant challenge in agriculture. However, the utilization of popular algorithms such as vision transformers and convolutional neural networks has often fallen short in numerous agricultural classification tasks owing to the scarcity of extensively labelled data and the reliance on pretrained models trained on generic datasets. To address this, our study details the pretraining of ViTs using 224,228 agricultural images, employing masked image modeling for preprocessing. The pretrained model was then fine-tuned on three independent agricultural classification datasets and performed better than state-of-the-art methods. For example, our method achieved the highest accuracy rates of 76.18%, 98.49%, and 88.56% on IP102, DeepWeeds, and Tsinghua Dogs datasets, respectively. This enhancement in accuracy can be attributed to the robust modeling strategy we have developed through extensive experimentation with the pretrained MIM model. Our modeling strategy encompasses employing advanced MIM models, leveraging the histogram of oriented gradient features as the reconstruction target, and selecting an appropriate mask ratio. We hope that this research will prompt the application of self-supervised learning techniques, represented by the MIM model, to a wide range of agricultural image-related tasks in the future.

关键词： Self-supervised learning masked image modeling agricultural classification tasks vision transformer histogram of oriented gradients feature

来源：评论

学校读者我要写书评

暂无评论

Disjoint Masking With Joint Distillation for Efficient masked image modeling

引用

IEEE TRANSACTIONS ON MULTIMEDIA 2024年 26卷 3077-3087页

作者： Ma, Xin Liu, Chang Xie, Chunyu Ye, Long Deng, Yafeng Ji, Xiangyang Commun Univ China Sch Informat & Commun Engn Beijing 100024 Peoples R China Tsinghua Univ Dept Automat Beijing 100084 Peoples R China 360 AI Res Beijing 100015 Peoples R China

masked image modeling (MIM) has shown great promise for self-supervised learning (SSL) yet been criticized for learning inefficiency. We believe the insufficient utilization of training signals should be responsible. To alleviate this issue, we introduce a conceptually simple yet learning-efficient MIM training scheme, termed Disjoint Masking with Joint Distillation (DMJD). For disjoint masking (DM), we sequentially sample multiple masked views per image in a mini-batch with the disjoint regulation to raise the usage of tokens for reconstruction in each image while keeping the masking rate of each view. For joint distillation (JD), we adopt a dual branch architecture to respectively predict invisible (masked) and visible (unmasked) tokens with superior learning targets. Rooting in orthogonal perspectives for training efficiency improvement, DM and JD cooperatively accelerate the training convergence yet not sacrificing the model generalization ability. Concretely, DM can train ViT with less effective training epochs (at most 3.7 chi less time-consuming) to report competitive performance. With JD, our DMJD clearly improves the linear probing classification accuracy, up to 3.4$%. On fine-grained downstream tasks like semantic segmentation, object detection, etc., our DMJD also presents superior generalization compared with state-of-the-art SSL methods.

关键词： Disjoint masking joint distillation masked image modeling self-supervised learning and training efficiency

来源：评论

学校读者我要写书评

暂无评论

autoSMIM: Automatic Superpixel-Based masked image modeling for Skin Lesion Segmentation

引用

IEEE TRANSACTIONS ON MEDICAL IMAGING 2023年第12期42卷 3501-3511页

作者： Wang, Zhonghua Lyu, Junyan Tang, Xiaoying Southern Univ Sci & Technol Dept Elect & Elect Engn Shenzhen 518055 Guangdong Peoples R China Southern Univ Sci & Technol Jiaxing Res Inst Jiaxing 314011 Zhejiang Peoples R China Univ Queensland Queensland Brain Inst St Lucia Qld 4072 Australia

Skin lesion segmentation from dermoscopic images plays a vital role in early diagnoses and prognoses of various skin diseases. However, it is a challenging task due to the large variability of skin lesions and their blurry boundaries. Moreover, most existing skin lesion datasets are designed for disease classification, with relatively fewer segmentation labels having been provided. To address these issues, we propose a novel automatic superpixel-based masked image modeling method, named autoSMIM, in a self-supervised setting for skin lesion segmentation. It explores implicit image features from abundant unlabeled dermoscopic images. autoSMIM begins with restoring an input image with randomly masked superpixels. The policy of generating and masking superpixels is then updated via a novel proxy task through Bayesian Optimization. The optimal policy is subsequently used for training a new masked image modeling model. Finally, we finetune such a model on the downstream skin lesion segmentation task. Extensive experiments are conducted on three skin lesion segmentation datasets, including ISIC 2016, ISIC 2017, and ISIC 2018. Ablation studies demonstrate the effectiveness of superpixel-based masked image modeling and establish the adaptability of autoSMIM. Comparisons with state-of-the-art methods show the superiority of our proposed autoSMIM. The source code is available at https://***/Wzhjerry/autoSMIM

关键词： Skin Lesions image segmentation Task analysis Data models Semantics Feature extraction Skin lesion segmentation self-supervised learning masked image modeling superpixel

来源：评论

学校读者我要写书评

暂无评论

Hybrid masked image modeling for 3D Medical image Segmentation

引用

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 2024年第4期28卷 2115-2125页

作者： Xing, Zhaohu Zhu, Lei Yu, Lequan Xing, Zhiheng Wan, Liang Tianjin Univ Med Sch Tianjin 300070 Peoples R China Hong Kong Univ Sci & Technol Guangzhou 510530 Peoples R China Hong Kong Univ Sci & Technol Henan Key Lab Imaging & Intelligent Proc Guangzhou 510530 Peoples R China Univ Hong Kong Hong Kong Peoples R China Tianjin Univ Haihe Hosp Tianjin 300072 Peoples R China Tianjin Univ Coll Comp & Intelligence Tianjin Peoples R China

masked image modeling (MIM) with transformer backbones has recently been exploited as a powerful self-supervised pre-training technique. The existing MIM methods adopt the strategy to mask random patches of the image and reconstruct the missing pixels, which only considers semantic information at a lower level, and causes a long pre-training time. This paper presents HybridMIM, a novel hybrid self-supervised learning method based on masked image modeling for 3D medical image segmentation. Specifically, we design a two-level masking hierarchy to specify which and how patches in sub-volumes are masked, effectively providing the constraints of higher level semantic information. Then we learn the semantic information of medical images at three levels, including: 1) partial region prediction to reconstruct key contents of the 3D image, which largely reduces the pre-training time burden (pixel-level);2) patch-masking perception to learn the spatial relationship between the patches in each sub-volume (region-level);and 3) drop-out-based contrastive learning between samples within a mini-batch, which further improves the generalization ability of the framework (sample-level). The proposed framework is versatile to support both CNN and transformer as encoder backbones, and also enables to pre-train decoders for image segmentation. We conduct comprehensive experiments on five widely-used public medical image segmentation datasets, including BraTS2020, BTCV, MSD Liver, MSD Spleen, and BraTS2023. The experimental results show the clear superiority of HybridMIM against competing supervised methods, masked pre-training approaches, and other self-supervised methods, in terms of quantitative metrics, speed performance and qualitative observations.

关键词： masked image modeling 3D medical image segmentation Self-supervised learning

来源：评论

学校读者我要写书评

暂无评论

PMIM: generating high-resolution air pollution data via masked image modeling

引用

JOURNAL OF VISUALIZATION 2024年第3期27卷 383-399页

作者： Wang, Mengyu Bi, Chongke Yang, Lu Qiu, Xiaobin Li, Yunlong Yu, Ce Tianjin Univ Tianjin Peoples R China Tianjin Univ Technol Tianjin Peoples R China Tianjin Key Lab Ocean Meteorol Tianjin Peoples R China Natl SuperComp Ctr Tianjin Tianjin Peoples R China

Air pollution data provides important information on air quality, which can be used to assess the impact of atmospheric pollution on human health, the environment, and the economy, as well as to develop corresponding policies and measures to reduce pollutant emissions and improve air quality. In this paper, we propose a novel approach to improve the resolution of meteorological data via masked image modeling (PMIM) to generate high-resolution air pollution data. In order to apply the image masking modeling to process air pollution data, we convert the data format and use radial basis function visualization to generate smooth distribution maps of air pollution data. To generate high-resolution air pollution data, we design several different masking strategies and use the masked image modeling to simulate the reconstruction process from low-resolution grid data to high-resolution grid data, obtaining the reconstructed high-resolution grid images. Finally, we use the mapping relationship between the pixel colors of the reconstructed images and the air pollution data to generate high-resolution air pollution concentration data. In order to verify the effectiveness of the proposed method, we conduct comparative experiments using different masking strategies and test air pollution data of different resolutions. The results show that our method has good applicability and effectiveness in different situations.

关键词： masked image modeling Mask strategy Air pollution Visualization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：