Distributed Learning with Sparse Communications by Identification

作     者：Grishchenko, Dmitry Iutzeler, Franck Malick, Jerome Amini, Massih-Reza 

作者机构：Univ Grenoble Alpes F-38000 Grenoble France CNRS LJK F-38401 St Martin Dheres France 

出 版 物：《SIAM JOURNAL ON MATHEMATICS OF DATA SCIENCE》 

年 卷 期：2021年第3卷第2期

页      面：715-735页

核心收录：

基　　金：IDEX Universite Grenoble Alpes IRS PGMO (Gaspard Monge Program for Optimization and Operations Research) MIAI @ Grenoble Alpes [ANR-19-P3IA-0003] ANR JCJC project STROLL [ANR19-CE23-0008] 

主　　题：optimization asynchronous algorithms structure-promoting regularization proximal algorithms 

摘      要：In distributed optimization for large-scale learning, a major performance limitation stems from the communications between the different entities. To the extent that computations are performed by workers on local data while a coordinator machine coordinates their updates to minimize a global loss, we present an asynchronous optimization algorithm that efficiently reduces the communications between the coordinator and workers. This reduction comes from a random sparsification of the local updates. We show that this algorithm converges linearly in the strongly convex case and also identifies optimal strongly sparse solutions. We further exploit this identification to propose an automatic dimension reduction, aptly sparsifying all exchanges between coordinator and workers.