Communication-Efficient Federated Learning via Quantized Compressed Sensing

作     者：Oh, Yongjeong Lee, Namyoon Jeon, Yo-Seb Poor, H. Vincent 

作者机构：POSTECH Dept Elect Engn Pohang 37673 South Korea Korea Univ Sch Elect Engn Seoul 02841 South Korea Princeton Univ Dept Elect & Comp Engn Princeton NJ 08544 USA 

出 版 物：《IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS》 (IEEE Trans. Wireless Commun.)

年 卷 期：2023年第22卷第2期

页      面：1087-1100页

核心收录：

学科分类：0810[工学-信息与通信工程] 0808[工学-电气工程] 08[工学] 

基　　金：National Research Foundation of Korea (NRF) - Korea Government (MSIT) [2022R1C1C1010074, 2020R1C1C1013381, 2022R1A5A1027646] BK21 FOUR Program 

主　　题：Federated learning quantized compressed sensing distributed stochastic gradient descent gradient compres-sion gradient reconstruction 

摘      要：In this paper, we present a communication-efficient federated learning framework inspired by quantized compressed sensing. The presented framework consists of gradient compression for wireless devices and gradient reconstruction for a parameter server (PS). Our strategy for gradient compression is to sequentially perform block sparsification, dimensional reduction, and quantization. By leveraging both dimension reduction and quantization, our strategy can achieve a higher compression ratio than one-bit gradient compression. For accurate aggregation of local gradients from the compressed signals, we put forth an approximate minimum mean square error (MMSE) approach for gradient reconstruction using the expectation-maximization generalized-approximate-message-passing (EM-GAMP) algorithm. Assuming Bernoulli Gaussian-mixture prior, this algorithm iteratively updates the posterior mean and variance of local gradients from the compressed signals. We also present a low-complexity approach for the gradient reconstruction. In this approach, we use the Bussgang theorem to aggregate local gradients from the compressed signals, then compute an approximate MMSE estimate of the aggregated gradient using the EM-GAMP algorithm. We also provide a convergence rate analysis of the presented framework. Using the MNIST dataset, we demonstrate that the presented framework achieves almost identical performance with the case that performs no compression, while significantly reducing communication overhead for federated learning.