Communication bandwidth is a bottleneck in distributed machine learning, and limits the system scalability. The transmission of gradients often dominates the communication in distributed SGD. One promising technique i...
详细信息
ISBN:
(纸本)9783030185794;9783030185787
Communication bandwidth is a bottleneck in distributed machine learning, and limits the system scalability. The transmission of gradients often dominates the communication in distributed SGD. One promising technique is using the gradient compression to reduce the communication cost. Recently, many approaches have been developed for the deep neural networks. However, they still suffer from the high memory cost, slow convergence and serious staleness problems over sparse high-dimensional models. In this work, we propose Sparse Gradient Compression (SGC) to efficiently train both the sparse models and the deep neural networks. SGC uses momentum approximation to reduce the memory cost with negligible accuracy degradation. Then it improves the accuracy with long-term gradient compensation, which maintains global momentum to make up for the information loss caused by the approximation. Finally, to alleviate the staleness problem, SGC updates model weight with the accumulation of delayed gradients at local, called local update technique. The experiments over the sparse high-dimensional models and deep neural networks indicate that SGC can compress 99.99% gradients for every iteration without performance degradation, and saves the communication cost up to 48x.
The physical power infrastructure is moving from the centralized structure to the distributed structure for enabling integration of distributed energy resources. Due to the large number of distributed energy resources...
详细信息
The physical power infrastructure is moving from the centralized structure to the distributed structure for enabling integration of distributed energy resources. Due to the large number of distributed energy resources, optimal resource allocation is an important and challenging problem. To solve this problem, a distributed and fast economic dispatch algorithm is provided to share the power generation task in an optimized fashion among a set of distributed energy resources, which can address both generation-demand equality and generation capacity inequality constraints. Different from most existing economic dispatch algorithms, the finite-time convergence to the optimal value is achieved, which makes more sense in real applications. Several case studies are discussed and tested to validate the proposed methods.
In recent years, due to the emergence of Big Data (terabytes or petabytes) and Big Model (tens of billions of parameters), there has been an ever-increasing need of parallelizing machine learning (ML) algorithms in bo...
详细信息
ISBN:
(纸本)9781450348874
In recent years, due to the emergence of Big Data (terabytes or petabytes) and Big Model (tens of billions of parameters), there has been an ever-increasing need of parallelizing machine learning (ML) algorithms in both academia and industry. Although there are some existing distributed computing systems, such as Hadoop and Spark, for parallelizing ML algorithms, they only provide synchronous and coarse-grained operators (e.g., Map, Reduce, and Join, etc.), which may hinder developers from implementing more efficient algorithms. This motivated us to design a universal distributed platform termed KunPeng, that combines both distributed systems and parallel optimizationalgorithms to deal with the complexities that arise from large-scale ML. Specifically, KunPeng not only encapsulates the characteristics of data/model parallelism, load balancing, model sync-up, sparse representation, industrial fault-tolerance, etc., but also provides easy-to-use interface to empower users to focus on the core ML logics. Empirical results on terabytes of real datasets with billions of samples and features demonstrate that, such a design brings compelling performance improvements on ML programs ranging from Follow-the-Regularized-Leader Proximal algorithm to Sparse Logistic Regression and Multiple Additive Regression Trees. Furthermore, KunPeng's encouraging performance is also shown for several real-world applications including the Alibaba's Double 11 Online Shopping Festival and Ant Financial's transaction risk estimation.
This paper presents a distributed transmission expansion planning (TEP) approach for multi-area power systems. A local TEP is formulated for each area with respect to local characteristics of that area as well as powe...
详细信息
ISBN:
(纸本)9781509051670
This paper presents a distributed transmission expansion planning (TEP) approach for multi-area power systems. A local TEP is formulated for each area with respect to local characteristics of that area as well as power flow in existing and candidate tie-lines that interconnect the area with its neighbors. Based on the auxiliary problem principle, a distributed decisionmaking algorithm is presented to coordinates the local TEP problems and find an overall solution for the entire power system. Limited information, only related to the tie-lines and not local lines, is exchanged between transmission areas, so that information privacy of the planners is respected. Numerical results on the Garver Test system show the effectiveness of the proposed distributed TEP algorithm.
In smart grid, virtual power plant (VPP) is a novel solution to distributed Energy Resources (DER) in the power grid. Same for traditional power plants, VPP should provide electricity generation to power grid accordin...
详细信息
In smart grid, virtual power plant (VPP) is a novel solution to distributed Energy Resources (DER) in the power grid. Same for traditional power plants, VPP should provide electricity generation to power grid according to the command electricity generation coming from power grid control center. Different from traditional power plants, VPP controls geographically distributed DER agents, which aggregate various local DER, such as small-scale engines, turbines or storages. Moreover, the number of DER Agents in VPP probably changes in pace with DER Agents plug in and plug out. In this paper, a distributed optimization algorithm is projected to solve VPP bi-level optimal dispatch considering uncertain agents number. Simulation results show that the proposed algorithm can effectively solve bi-level optimal dispatch with changing lower agents. (C) 2015 Elsevier B.V. All rights reserved.
Quadratic optimization problems appear in several interesting estimation, learning and control tasks. To solve these problems in peer-to-peer networks it is necessary to design distributed optimization algorithms supp...
详细信息
ISBN:
(纸本)9783952426937
Quadratic optimization problems appear in several interesting estimation, learning and control tasks. To solve these problems in peer-to-peer networks it is necessary to design distributed optimization algorithms supporting directed, asynchronous and unreliable communication. This paper addresses this requirement by extending a promising distributed convex optimizationalgorithm, known as Newton-Raphson consensus, and originally designed for static and undirected communication. Specifically, we modify this algorithm so that it can cope with asynchronous, broadcast and unreliable lossy links, and prove that the optimization strategy correctly converge to the global optimum when the local cost functions are quadratic. We then support the intuition that this robustified algorithm converges to the true optimum also for general convex problems with dedicated numerical simulations.
暂无评论