检索结果-内蒙古大学图书馆

distributed stochastic gradient descent With Compressed and Skipped Communication

IEEE ACCESS 2023年 11卷 99836-99846页

作者： Phuong, Tran Thi Phong, Le Trieu Fukushima, Kazuhide KDDI Res Inc Saitama 3568502 Japan Natl Inst Informat & Commun Technol NICT Tokyo 1848795 Japan

This paper introduces CompSkipDSGD, a new algorithm for distributed stochastic gradient descent that aims to improve communication efficiency by compressing and selectively skipping communication. In addition to compression, CompSkipDSGD allows both workers and the server to skip communication in any iteration of the training process and reserve it for future iterations without significantly decreasing testing accuracy. Our experimental results on the large-scale ImageNet dataset demonstrate that CompSkipDSGD can save hundreds of gigabytes of communication while maintaining similar levels of accuracy compared to state-of-the-art algorithms. The experimental results are supported by a theoretical analysis that demonstrates the convergence of CompSkipDSGD under established assumptions. Overall, CompSkipDSGD could be useful for reducing communication costs in distributed deep learning and enabling the use of large-scale datasets and models in complex environments.

关键词： Compressed sensing gradient methods Compressed and skipped communication distributed stochastic gradient descent deep learning

来源：评论

学校读者我要写书评

暂无评论

Communication-Efficient Federated Learning via Quantized Compressed Sensing

引用

IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS 2023年第2期22卷 1087-1100页

作者： Oh, Yongjeong Lee, Namyoon Jeon, Yo-Seb Poor, H. Vincent POSTECH Dept Elect Engn Pohang 37673 South Korea Korea Univ Sch Elect Engn Seoul 02841 South Korea Princeton Univ Dept Elect & Comp Engn Princeton NJ 08544 USA

In this paper, we present a communication-efficient federated learning framework inspired by quantized compressed sensing. The presented framework consists of gradient compression for wireless devices and gradient reconstruction for a parameter server (PS). Our strategy for gradient compression is to sequentially perform block sparsification, dimensional reduction, and quantization. By leveraging both dimension reduction and quantization, our strategy can achieve a higher compression ratio than one-bit gradient compression. For accurate aggregation of local gradients from the compressed signals, we put forth an approximate minimum mean square error (MMSE) approach for gradient reconstruction using the expectation-maximization generalized-approximate-message-passing (EM-GAMP) algorithm. Assuming Bernoulli Gaussian-mixture prior, this algorithm iteratively updates the posterior mean and variance of local gradients from the compressed signals. We also present a low-complexity approach for the gradient reconstruction. In this approach, we use the Bussgang theorem to aggregate local gradients from the compressed signals, then compute an approximate MMSE estimate of the aggregated gradient using the EM-GAMP algorithm. We also provide a convergence rate analysis of the presented framework. Using the MNIST dataset, we demonstrate that the presented framework achieves almost identical performance with the case that performs no compression, while significantly reducing communication overhead for federated learning.

关键词： Federated learning quantized compressed sensing distributed stochastic gradient descent gradient compres-sion gradient reconstruction

来源：评论

学校读者我要写书评

暂无评论

NOVEL gradient SPARSIFICATION ALGORITHM VIA BAYESIAN INFERENCE 34

NOVEL GRADIENT SPARSIFICATION ALGORITHM VIA BAYESIAN INFEREN...

引用

34th International Workshop on Machine Learning for Signal Processing

作者： Bereyhi, Ali Liang, Ben Boudreau, Gary Afana, Ali Univ Toronto ECE Dept Toronto ON Canada Ericsson Canada Ottawa ON Canada

ISBN: (纸本)9798350372267;9798350372250

Error accumulation is an essential component of the TOP-k sparsification method in distributed gradient descent. It implicitly scales the learning rate and prevents the slow-down of lateral movement, but it can also deteriorate convergence. This paper proposes a novel sparsification algorithm called regularized TOP-k (REGTOP-k) that controls the learning rate scaling of error accumulation. The algorithm is developed by looking at the gradient sparsification as an inference problem and determining a Bayesian optimal sparsification mask via maximum-a-posteriori estimation. It utilizes past aggregated gradients to evaluate posterior statistics, based on which it prioritizes the local gradient entries. Numerical experiments with ResNet-18 on CIFAR-10 show that at 0.1% sparsification, REGTOP-k achieves about 8% higher accuracy than standard TOP-k.

关键词： gradient sparsification TOP-k algorithm Bayesian inference distributed stochastic gradient descent communication-efficient distributed learning

来源：评论

学校读者我要写书评

暂无评论

distributed differentially-private learning with communication efficiency

引用

JOURNAL OF SYSTEMS ARCHITECTURE 2022年 128卷

作者： Phuong, Tran Thi Phong, Le Trieu Meiji Univ Tokyo Kanagawa 2148571 Japan Natl Inst Informat & Commun Technol NICT Tokyo 1848795 Japan

In this paper, we propose a new algorithm for learning over distributed data such as in the IoT environment, in a privacy-preserving way. Our algorithm is a differentially private variant of distributed synchronous stochastic gradient descent method with multiple workers and one parameter server, and has the following two features: (1) each distributed worker only needs to send as small as O(1) gradients in each iteration, so that the communication from worker to server is modest;(2) the dataset of each worker is protected quantitatively by differential privacy. We mathematically prove the convergence of our algorithm, and experimentally verify that it converges and reaches standard testing results on a benchmark dataset, while simultaneously maintaining reasonable privacy budgets. Our results address two equally important issues in the IoT environment, communication efficiency and differential privacy, and potentially help reducing the tension caused by the issues.

关键词： Data privacy in IoT distributed stochastic gradient descent Communication efficiency

来源：评论

学校读者我要写书评

暂无评论

Communication-Efficient distributed SGD Using Random Access for Over-the-Air Computation

IEEE JOURNAL ON SELECTED AREAS IN INFORMATION THEORY

引用

IEEE JOURNAL ON SELECTED AREAS IN INFORMATION THEORY 2022年第2期3卷 206-216页

作者： Choi, Jinho Deakin Univ Sch Informat Technol Geelong Vic 3220 Australia

In this paper, we study communication-efficient distributed stochastic gradient descent (SGD) with data sets of users distributed over a certain area and communicating through wireless channels. Since the time for one iteration in the proposed approach is independent of the number of users, it is well-suited to scalable distributed SGD. Furthermore, since the proposed approach is based on preamble-based random access, which is widely adopted for machine-type communication (MTC), it can be easily employed for training models with a large number of devices where MTC is used for their connectivity. For fading channel, we show that noncoherent combining can be used. As a result, no channel state information (CSI) estimation is required. For over-the-air computation, the proposed approach takes advantage of random access, where the access probability is to encode the norm of local gradient vector (without using any additional bits) and the preamble is to encode the quantized normalized gradient vector. From analysis and simulation results, we can confirm that the proposed approach is not only scalable, but also provides improved performance as the number of devices increases. From analysis and simulation results, we can confirm that the proposed approach is not only scalable, but also provides improved performance as the number of devices increases.

关键词： distributed stochastic gradient descent random access Internet-of-Things

来源：评论

学校读者我要写书评

暂无评论

Sketch-fusion: A gradient compression method with multi-layer fusion for communication-efficient distributed training

引用

JOURNAL OF PARALLEL AND distributed COMPUTING 2024年 185卷

作者： Dai, Lingfei Gong, Luqi An, Zhulin Xu, Yongjun Diao, Boyu Chinese Acad Sci Inst Comp Technol Beijing Peoples R China Univ Chinese Acad Sci Coll Comp Sci Beijing Peoples R China

gradient compression is an effective technique for improving the efficiency of distributed training. However, introducing gradient compression can reduce model accuracy and training efficiency. Furthermore, we also find that using a layer-wise gradient compression algorithm would lead to significant compression and communication overhead, which can negatively impact the scaling efficiency of the distributed training system. To address these issues, we propose a new method called Sketch-Fusion SGD, which leverages the Count-Sketch data structure to enhance the scalability and training speed of distributed deep learning systems. Moreover, our method employs LayerFusion to optimize gradient compression algorithms' scalability and convergence efficiency by formulating an optimal multi-layer fusion strategy without introducing extra hyperparameters. We evaluate our method on a cluster of 16 GPUs and demonstrate that it can improve training efficiency by up to 18.6% without compromising the model's accuracy. In addition, we find that applying our LayerFusion algorithm to other gradient compression methods improved their scalability by up to 2.87x.

关键词： gradient compression Multi-layer fusion distributed stochastic gradient descent Deep learning training

来源：评论

学校读者我要写书评

暂无评论

A Random Access based Approach to Communication-Efficient distributed SGD

A Random Access based Approach to Communication-Efficient Di...

引用

IEEE International Conference on Communications (ICC)

作者： Choi, Jinho Deakin Univ Sch Informat Technol Geelong Vic Australia

ISBN: (数字)9781538683477

ISBN: (纸本)9781538683477

关键词： distributed stochastic gradient descent Random Access Internet-of-Things

来源：评论

学校读者我要写书评

暂无评论

A Compressive Sensing Approach for Federated Learning Over Massive MIMO Communication Systems

引用

IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS 2021年第3期20卷 1990-2004页

作者： Jeon, Yo-Seb Amiri, Mohammad Mohammadi Li, Jun Poor, H. Vincent POSTECH Dept Elect Engn Pohang 37673 South Korea Princeton Univ Dept Elect Engn Princeton NJ 08544 USA Nanjing Univ Sci & Technol Sch Elect & Opt Engn Nanjing 210094 Peoples R China Natl Res Tomsk Polytech Univ Sch Comp Sci & Robot Tomsk 634050 Russia

Federated learning is a privacy-preserving approach to train a global model at a central server by collaborating with wireless devices, each with its own local training data set. In this paper, we present a compressive sensing approach for federated learning over massive multiple-input multiple-output communication systems in which the central server equipped with a massive antenna array communicates with the wireless devices. One major challenge in system design is to reconstruct local gradient vectors accurately at the central server, which are computed-and-sent from the wireless devices. To overcome this challenge, we first establish a transmission strategy to construct sparse transmitted signals from the local gradient vectors at the devices. We then propose a compressive sensing algorithm enabling the server to iteratively find the linear minimum-mean-square-error (LMMSE) estimate of the transmitted signal by exploiting its sparsity. We also derive an analytical threshold for the residual error at each iteration, to design the stopping criterion of the proposed algorithm. We show that for a sparse transmitted signal, the proposed algorithm requires less computationally complexity than LMMSE. Simulation results demonstrate that the presented approach outperforms conventional linear beamforming approaches and reduces the performance gap between federated learning and centralized learning with perfect reconstruction.

关键词： Federated learning distributed stochastic gradient descent massive multiple-input multiple-output (MIMO) compressive sensing multi-antenna technique

来源：评论

学校读者我要写书评

暂无评论

MG-WFBP: Merging gradients Wisely for Efficient Communication in distributed Deep Learning

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed SYSTEMS 2021年第8期32卷 1903-1917页

作者： Shi, Shaohuai Chu, Xiaowen Li, Bo Hong Kong Univ Sci & Technol Dept Comp Sci & Engn Kowloon Hong Kong Peoples R China Hong Kong Baptist Univ Dept Comp Sci Kowloon Hong Kong Peoples R China

distributed synchronous stochastic gradient descent has been widely used to train deep neural networks (DNNs) on computer clusters. With the increase of computational power, network communications generally limit the system scalability. Wait-free backpropagation (WFBP) is a popular solution to overlap communications with computations during the training process. In this article, we observe that many DNNs have a large number of layers with only a small amount of data to be communicated at each layer in distributed training, which could make WFBP inefficient. Based on the fact that merging some short communication tasks into a single one can reduce the overall communication time, we formulate an optimization problem to minimize the training time in pipelining communications and computations. We derive an optimal solution that can be solved efficiently without affecting the training performance. We then apply the solution to propose a distributed training algorithm named merged-gradient WFBP (MG-WFBP) and implement it in two platforms Caffe and PyTorch. Extensive experiments in three GPU clusters are conducted to verify the effectiveness of MG-WFBP. We further exploit trace-based simulations of 4 to 2048 GPUs to explore the potential scaling efficiency of MG-WFBP. Experimental results show that MG-WFBP achieves much better scaling performance than existing methods.

关键词： Training Backpropagation Hardware Graphics processing units Tensors Neural networks Data models Deep learning GPU distributed stochastic gradient descent gradient communication merged-gradient

来源：评论

学校读者我要写书评

暂无评论

Partitioning Sparse Deep Neural Networks for Scalable Training and Inference 21

Partitioning Sparse Deep Neural Networks for Scalable Traini...

引用

35th ACM International Conference on Supercomputing (ICS)

作者： Demirci, Gunduz Vehbi Ferhatosmanoglu, Hakan Univ Warwick Coventry W Midlands England

ISBN: (纸本)9781450383356

The state-of-the-art deep neural networks (DNNs) have significant computational and data management requirements. The size of both training data and models continue to increase. Sparsification and pruning methods are shown to be effective in removing a large fraction of connections in DNNs. The resulting sparse networks present unique challenges to further improve the computational efficiency of training and inference in deep learning. Both the feed-forward (inference) and backpropagation steps in stochastic gradient descent (SGD) algorithm for training sparse DNNs involve consecutive sparse matrix-vector multiplications (SpMVs). We first introduce a distributed-memory parallel SpMV-based solution for the SGD algorithm to improve its scalability. The parallelization approach is based on row-wise partitioning of weight matrices that represent neuron connections between consecutive layers. We then propose a novel hypergraph model for partitioning weight matrices to reduce the total communication volume and ensure computational load-balance among processors. Experiments performed on sparse DNNs demonstrate that the proposed solution is highly efficient and scalable. By utilizing the proposed matrix partitioning scheme, the performance of our solution is further improved significantly.

关键词： Scalable Deep Learning Sparse Deep Neural Networks distributed stochastic gradient descent Hypergraph Partitioning Sparse Matrix Vector Multiplication

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：