检索结果-内蒙古大学图书馆

Multimodal Feature-Guided Pretraining for RGB-T Perception

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING 2024年 17卷 16041-16050页

作者： Ouyang, Junlin Jin, Pengcheng Wang, Qingwang Kunming Univ Sci & Technol Fac Informat Engn & Automat Kunming 650500 Peoples R China

Wide-range multiscale object detection for multispectral scene perception from a drone perspective is challenging. Previous RGB-T perception methods directly use backbone pretrained on RGB for thermal infrared feature extraction, leading to unexpected domain shift. We propose a novel multimodal feature-guided masked reconstruction pretraining method, named M2FP, aimed at learning transferable representations for drone-based RGB-T environmental perception tasks without domain bias. This article includes two key innovations as follows. 1) We design a cross-modal feature interaction module in M2FP, which encourages modality-specific backbones to actively learn cross-modal feature representations and avoid modality bias issues. 2) We design a global-aware feature interaction and fusion module suitable for various downstream tasks, which enhances the model's environmental perception from a global perspective in wide-range drone-based scenes. We fine-tune M2FP on the drone-based object detection dataset (DroneVehicle) and semantic segmentation dataset (Kust4K). On these two tasks, compared to the second-best methods, M2FP achieves state-of-the-art performance, with an improvement of 1.8% in mean average precision and 0.9% in mean intersection over union, respectively.

关键词： Feature extraction Task analysis Image reconstruction Object detection Semantic segmentation Pipelines Visualization masked autoencoder multimodal object detection semantic segmentation unmanned aerial vehicle (UAV) remote sensing

来源：评论

学校读者我要写书评

暂无评论

ViT-UperNet: a hybrid vision transformer with unified-perceptual-parsing network for medical image segmentation

引用

COMPLEX & INTELLIGENT SYSTEMS 2024年第3期10卷 3819-3831页

作者： Ruiping, Yang Kun, Liu Shaohua, Xu Jian, Yin Zhen, Zhang Shandong Univ Sci & Technol Coll Econ & Management Qianwangang Rd Qingdao 266590 Shandong Peoples R China Shandong Univ Sci & Technol Coll Comp Sci & Engn Qianwangang Rd Qingdao 266590 Shandong Peoples R China

The existing image semantic segmentation models have low accuracy in detecting tiny targets or multi-targets at overlapping regions. This work proposes a hybrid vision transformer with unified-perceptual-parsing network (ViT-UperNet) for medical image segmentation. A self-attention mechanism is embedded in a vision transformer to extract multi-level features. The image features are extracted hierarchically from low to high dimensions using 4 groups of Transformer blocks with different numbers. Then, it uses a unified-perceptual-parsing network based on a feature pyramid network (FPN) and a pyramid pooling module (PPM) for the fusion of multi-scale contextual features and semantic segmentation. FPN can naturally use hierarchical features, and generate strong semantic information on all scales. PPM can better use the global prior knowledge to understand complex scenes, and extract features with global context information to improve segmentation results. In the training process, a scalable self-supervised learner named masked autoencoder is used for pre-training, which strengthens the visual representation ability and improves the efficiency of the feature learning. Experiments are conducted on cardiac magnetic resonance image segmentation where the left and right atrium and ventricle are selected for segmentation. The pixels accuracy is 93.85%, the Dice coefficient is 92.61% and Hausdorff distance is 11.16, which are improved compared with the other methods. The results show the superiority of Vit-UperNet in medical images segmentation, especially for the low-recognition and serious-occlusion targets.

关键词： Medical image segmentation Deep learning Self-attention mechanism masked autoencoder

来源：评论

学校读者我要写书评

暂无评论

Slow feature-constrained decomposition autoencoder: Application to process anomaly detection and localization

引用

INTERNATIONAL JOURNAL OF ADAPTIVE CONTROL AND SIGNAL PROCESSING 2024年第0期

作者： Jia, Mingwei Jiang, Lingwei Hu, Junhao Liu, Yi Chen, Tao Zhejiang Univ Technol Inst Proc Equipment & Control Engn Hangzhou 310023 Peoples R China Univ Surrey Sch Chem & Chem Engn Guildford GU2 7XH Surrey England

Detecting anomalies in manufacturing processes is crucial for ensuring safety. However, noise significantly undermines the reliability of data-driven anomaly detection models. To address this challenge, we propose a slow feature-constrained decomposition autoencoder (SFC-DAE) for anomaly detection in noisy scenarios. Considering that the process can exhibit both long-term trends and periodic properties, the process data is decomposed into trends and cycles. The repetitive information is mitigated by slicing and randomly masking certain trends and cycles. Dependencies among slices are constructed to extract intrinsic information, while high-frequency noise is reduced using a slow feature-constrained loss. Anomalies are detected and localized through a reconstruction error strategy. The effectiveness of SFC-DAE is demonstrated using data from a sugar factory and a secure water treatment system.

关键词： anomaly detection masked autoencoder noisy data process industry time series decomposition

来源：评论

学校读者我要写书评

暂无评论

Content-based Graph Reconstruction for Cold-start Item Recommendation 47

Content-based Graph Reconstruction for Cold-start Item Recom...

引用

47th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

作者： Kim, Jinri Kim, Eungi Yeo, Kwangeun Jeon, Yujin Kim, Chanwoo Lee, Sewon Lee, Joonseok Seoul Natl Univ Seoul South Korea

ISBN: (纸本)9798400704314

Graph convolutions have been successfully applied to recommendation systems, utilizing high-order collaborative signals present in the user-item interaction graph. This idea, however, has not been applicable to the cold-start items, since cold nodes are isolated in the graph and thus do not take advantage of information exchange from neighboring nodes. Recently, there have been a few attempts to utilize graph convolutions on item-item or user-user attribute graphs to capture high-order collaborative signals for cold-start cases, but these approaches are still limited in that the item-item or user-user graph falls short in capturing the dynamics of user-item interactions, as their edges are constructed based on arbitrary and heuristic attribute similarity. In this paper, we introduce Content-based Graph Reconstruction for Cold-start item recommendation (CGRC), employing a masked graph autoencoder structure and multimodal contents to directly incorporate interaction-based high-order connectivity, applicable even in cold-start scenarios. To address the cold-start items directly on the interaction graph, our approach trains the model to reconstruct plausible user-item interactions from masked edges of randomly chosen cold items, simulating fresh items without connection to users. This strategy enables the model to infer potential edges for unseen cold-start nodes. Extensive experiments on real-world datasets demonstrate the superiority of our model.

关键词： Cold-start Recommendation Graph Neural Networks Multi-modal masked autoencoder

来源：评论

学校读者我要写书评

暂无评论

Image Retrieval Based on Vision Transformer and masked Learning

引用

Journal of Donghua University(English Edition) 2023年第5期40卷 539-547页

作者：李锋潘煌圣盛守祥王国栋 College of Computer Science and Technology Donghua UniversityShanghai 201620China Huafang Co. Ltd.Binzhou 256617China

Deep convolutional neural networks(DCNNs)are widely used in content-based image retrieval(CBIR)because of the advantages in image feature ***,the training of deep neural networks requires a large number of labeled data,which limits the ***-supervised learning is a more general approach in unlabeled scenarios.A method of fine-tuning feature extraction networks based on masked learning is *** autoencoders(MAE)are used in the fine-tune vision transformer(ViT)*** addition,the scheme of extracting image descriptors is *** encoder of the MAE uses the ViT to extract global features and performs self-supervised fine-tuning by reconstructing masked area *** method works well on category-level image retrieval datasets with marked improvements in instance-level *** the instance-level datasets Oxford5k and Paris6k,the retrieval accuracy of the base model is improved by 7%and 17%compared to that of the original model,respectively.

关键词： content-based image retrieval vision transformer masked autoencoder feature extraction

来源：评论

学校读者我要写书评

暂无评论

Semi-Supervised Multimodal Emotion Recognition with Expression MAE 23

Semi-Supervised Multimodal Emotion Recognition with Expressi...

引用

31st ACM International Conference on Multimedia (MM)

作者： Cheng, Zebang Lin, Yuxiang Chen, Zhaoru Li, Xiang Mao, Shuyi Zhang, Fan Ding, Daijun Zhang, Bowen Peng, Xiaojiang Shenzhen Technol Univ Shenzhen Peoples R China

ISBN: (纸本)9798400701085

The Multimodal Emotion Recognition (MER 2023) challenge aims to recognize emotion with audio, language, and visual signals, facilitating innovative technologies of affective computing. This paper presents our submission approach on the Semi-Supervised Learning Sub-Challenge (MER-SEMI). First, with large-scale unlabeled emotional videos, we train both image-based and video-based masked autoencoders to extract visual features, which termed as expression MAE (expMAE) for simplicity. The expMAE features are found to be largely complementary with other official baseline features. Second, since there is only a few labeled data, we use a classifier to generate pseudo labels for unlabeled videos which have high confidence for a certain category. In addition, we also explore several advanced large models for cross-feature extraction like CLIP, and apply factorized bilinear pooling (FBP) for multimodal feature fusion. Our methods finally achieved 88.55% in F1 score on MER-SEMI, ranking second place among all participating teams.

关键词： Multimodal Emotion Recognition Semi-Supervised Learning masked autoencoder

来源：评论

学校读者我要写书评

暂无评论

MART: masked Affective RepresenTation Learning via masked Temporal Distribution Distillation

MART: Masked Affective RepresenTation Learning via Masked Te...

引用

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

作者： Zhang, Zhicheng Zhao, Pancheng Park, Eunil Yang, Jufeng Nankai Univ Coll Comp Sci VCIP & TMCC & DISSec Tianjin Peoples R China Nankai Int Adv Res Inst SHENZHEN FUTIAN Shenzhen Peoples R China Sungkyunkwan Univ Coll Comp Seoul South Korea

ISBN: (纸本)9798350353006

Limited training data is a long-standing problem for video emotion analysis (VEA). Existing works leverage the power of large-scale image datasets for transferring while failing to extract the temporal correlation of affective cues in the video. Inspired by psychology research and empirical theory, we verify that the degree of emotion may vary in different segments of the video, thus introducing the sentiment complementary and emotion intrinsic among temporal segments. We propose an MAE-style method for learning robust affective representation of videos via masking, termed MART. First, we extract the affective cues of the lexicon and verify the extracted one by computing its matching score with video content, in terms of sentiment and emotion scores alongside the temporal dimension. Then, with the verified cues, we propose masked affective modeling to recover temporal emotion distribution. We present temporal affective complementary learning that pulls the complementary part and pushes the intrinsic one of masked multimodal features, where the constraint is set with cross-modal attention among features to mask the video and recover the degree of emotion among segments. Extensive experiments on five benchmarks show the superiority of our method in video sentiment analysis, video emotion recognition, multimodal sentiment analysis, and multimodal emotion recognition.

关键词： masked autoencoder Video Emotion Analysis

来源：评论

学校读者我要写书评

暂无评论

Graph Transformer for Recommendation 23

Graph Transformer for Recommendation

引用

46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)

作者： Li, Chaoliu Xia, Lianghao Ren, Xubin Ye, Yaowen Xu, Yong Huang, Chao South China Univ Technol Guangzhou Peoples R China Univ Hong Kong Hong Kong Peoples R China

ISBN: (纸本)9781450394086

This paper presents a novel approach to representation learning in recommender systems by integrating generative self-supervised learning with graph transformer architecture. We highlight the importance of high-quality data augmentation with relevant self-supervised pretext tasks for improving performance. Towards this end, we propose a newapproach that automates the self-supervision augmentation process through a rationale-aware generative SSL that distills informative user-item interaction patterns. The proposed recommender with Graph TransFormer (GFormer) that offers parameterized collaborative rationale discovery for selective augmentation while preserving global-aware user-item relationships. In GFormer, we allow the rationale-aware SSL to inspire graph collaborative filtering with task-adaptive invariant rationalization in graph transformer. The experimental results reveal that our GFormer has the capability to consistently improve the performance over baselines on different datasets. Several in-depth experiments further investigate the invariant rationale-aware augmentation from various aspects. The source code for this work is publicly available at: https://***/HKUDS/GFormer.

关键词： Recommendation Graph Transformer masked autoencoder

来源：评论

学校读者我要写书评

暂无评论

Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval

Contrastive masked auto-encoders based self-supervised hashi...

引用

IEEE International Conference on Multimedia and Expo (ICME)

作者： Wei, Rukai Cui, Heng Liu, Yu Xie, Yanzhao Hou, Yufeng Zhou, Ke HUST Wuhan Natl Lab Optoelect Wuhan Peoples R China HUST Sch Software Engn Wuhan Peoples R China HUST Sch Comp Sci & Technol Wuhan Hubei Peoples R China GZU Sch Comp Sci & Cyber Engn Guangzhou Peoples R China

ISBN: (纸本)9798350390155;9798350390162

Implementing cross-modal hashing between 2D images and 3D point-cloud data is a growing concern in real-world retrieval systems. Simply applying existing cross-modal approaches to this new task fails to adequately capture latent multi-modal semantics and effectively bridge the modality gap between 2D and 3D. To address these issues without relying on hand-crafted labels, we propose contrastive masked autoencoders based self-supervised hashing (CMAH) for retrieval between images and point-cloud data. We start by contrasting 2D-3D pairs and explicitly constraining them into a joint Hamming space. This contrastive learning process ensures robust discriminability for the generated hash codes and effectively reduces the modality gap. Moreover, we utilize multi-modal auto-encoders to enhance the model's understanding of multi-modal semantics. By completing the masked image/point-cloud data modeling task, the model is encouraged to capture more localized clues. In addition, the proposed multi-modal fusion block facilitates fine-grained interactions among different modalities. Extensive experiments on three public datasets demonstrate that the proposed CMAH significantly outperforms all baseline methods.

关键词： 3D point-cloud data cross-modal hashing contrastive learning masked autoencoder retrieval

来源：评论

学校读者我要写书评

暂无评论

DUAL-BRANCH POLSAR IMAGE CLASSIFICATION BASED ON GRAPHMAE AND LOCAL FEATURE EXTRACTION

DUAL-BRANCH POLSAR IMAGE CLASSIFICATION BASED ON GRAPHMAE AN...

引用

IEEE International Geoscience and Remote Sensing Symposium (IGARSS)

作者： Wang, Yuchen Guo, Ziyi Bi, Haixia Hong, Danfeng Xu, Chen Xi An Jiao Tong Univ Sch Informat & Commun Engn Xian 710049 Peoples R China Chinese Acad Sci Aerosp Informat Res Inst Key Lab Digital Earth Sci Beijing 100094 Peoples R China Peng Cheng Lab Dept Math & Fundamental Res Shenzhen 518055 Peoples R China Xi An Jiao Tong Univ Sch Math & Stat Xian 710049 Peoples R China

ISBN: (纸本)9798350360332;9798350360325

The annotation of polarimetric synthetic aperture radar (PolSAR) images is a labor-intensive and time-consuming process. Therefore, classifying PolSAR images with limited labels is a challenging task in remote sensing domain. In recent years, self-supervised learning approaches have proven effective in PolSAR image classification with sparse labels. However, we observe a lack of research on generative selfsupervised learning in the studied task. Motivated by this, we propose a dual-branch classification model based on generative self-supervised learning in this paper. The first branch is a superpixel-branch, which learns superpixel-level polarimetric representations using a generative self-supervised graph masked autoencoder. To acquire finer classification results, a convolutional neural networks-based pixel-branch is further incorporated to learn pixel-level features. Classification with fused dual-branch features is finally performed to obtain the predictions. Experimental results on the benchmark Flevoland dataset demonstrate that our approach yields promising classification results.

关键词： PolSAR image classification graph neural networks masked autoencoder self-supervised learning convolutional neural networks

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：