USDRL: Unified Skeleton-Based Dense Representation Learning with Multi-Grained Feature Decorrelation

作     者：Weng, Wanjiang Wang, Hongsong Wang, Junbo He, Lei Xie, Guosen 

作者机构：School of Computer Science and Engineering Southeast University Nanjing210096 China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications Ministry of Education China School of Software Northwestern Polytechnical University Xi’an710072 China School of Vehicle and Mobility Tsinghua University Beijing China School of Computer Science and Engineering Nanjing University of Science and Technology Nanjing China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Contrastive Learning 

摘      要：Contrastive learning has achieved great success in skeleton-based representation learning recently. However, the prevailing methods are predominantly negative-based, necessitating additional momentum encoder and memory bank to get negative samples, which increases the difficulty of model training. Furthermore, these methods primarily concentrate on learning a global representation for recognition and retrieval tasks, while overlooking the rich and detailed local representations that are crucial for dense prediction tasks. To alleviate these issues, we introduce a Unified Skeleton-based Dense Representation Learning framework based on feature decorrelation, called USDRL, which employs feature decorrelation across temporal, spatial, and instance domains in a multi-grained manner to reduce redundancy among dimensions of the representations to maximize information extraction from features. Additionally, we design a Dense Spatio-Temporal Encoder (DSTE) to capture fine-grained action representations effectively, thereby enhancing the performance of dense prediction tasks. Comprehensive experiments, conducted on the benchmarks NTU-60, NTU-120, PKU-MMD I, and PKU-MMD II, across diverse downstream tasks including action recognition, action retrieval, and action detection, conclusively demonstrate that our approach significantly outperforms the current state-of-the-art (SOTA) approaches. Our code and models are available at https://***/wengwanjiang/USDRL. © 2024, CC BY.