Adapter-Enhanced Semantic Prompting for Continual Learning

作     者：Yin, Baocai Zhao, Ji Jiang, Huajie Hou, Ningning Hu, Yongli Beheshti, Amin Yang, Ming-Hsuan Qi, Yuankai 

作者机构：Beijing Key Laboratory of Multimedia and Intelligent Software Technology Beijing Institute of Artificial Intelligence Faculty of Information Technology Beijing University of Technology Beijing100124 China Department of Electrical Engineering and Computer Science University of California at Merced MercedCA95343 United States School of Computing Macquarie University SydneyNSW2109 Australia 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Contrastive Learning 

摘      要：Continual learning (CL) is essential for enabling models to adapt to dynamic data streams, with the primary challenge being the mitigation of catastrophic forgetting of previously acquired knowledge. Recent advancements have highlighted the potential of prompt-based CL methodologies, which have shown promise in preserving the knowledge embedded within pre-trained models, thereby addressing catastrophic forgetting. However, these methodologies are constrained by their focus on learning visual prompts for individual tasks, resulting in suboptimal generalization to unseen categories. To enhance the model’s ability to retain knowledge from prior tasks while simultaneously improving generalization to new tasks, we introduce a novel, lightweight framework for CL termed Adapter-Enhanced Semantic Prompting (AESP). This framework leverages class semantic information to augment visual features, thereby establishing relational links among different categories. This not only fortifies the retention of prior knowledge but also facilitates adaptation to new tasks. To seamlessly integrate visual and semantic information, we have developed a lightweight Adapter-enhanced Vision Transformer (ViT) architecture specifically designed for feature adaptation. Furthermore, we have implemented a robust Query-Key matching strategy to select the optimal task prompt pair, thereby enhancing the accuracy of final predictions. Comprehensive experiments conducted across three benchmark continual learning datasets demonstrate that our proposed framework outperforms several state-of-the-art approaches, showcasing its efficacy and robustness. Copyright © 2024, The Authors. All rights reserved.