Federated learning (FL) is an emerging distributed machine learning paradigm, which has shown great potential in collaborative learning with privacy preservation. However, FL clients usually have disparate system reso...
详细信息
ISBN:
(纸本)9798350339864
Federated learning (FL) is an emerging distributed machine learning paradigm, which has shown great potential in collaborative learning with privacy preservation. However, FL clients usually have disparate system resource capabilities (e.g., data, computation, and communication) for model training and aggregation, which can cause a series of system heterogeneity issues with performance degradation. To this end, we propose FedPKD, a prototype-based knowledge distillation framework for FL. FedPKD integrates knowledgedistillation and prototype learning with FL, which enables heterogeneous clients and the server to learn collaboratively, with different model architectures and resource capability adaptations. Specifically, FedPKD proposes to transfer dual knowledge of clients including the model output logits and prototypes to the server, and a prototype-based ensemble distillation mechanism is proposed to aggregate the logits and prototypes from clients, which can be used to train the server model with an unlabeled public dataset. The server model knowledge is then transferred back to clients to improve the performance of client models. Moreover, to improve learning performance and reduce communication overhead, we propose a prototype-based data filter mechanism to filter out the samples with low-quality knowledge. Extensive experiments under various settings demonstrate the superiority of FedPKD in learning performance and communication efficiency when compared to state-of-the-art benchmarks.
暂无评论