Stochastic gradient descent (SGD) is a widely adopted iterative method for optimizing differentiable objective functions. In this paper, we propose and discuss a novel approach to scale up SGD in applications involvin...
详细信息
Stochastic gradient descent (SGD) is a widely adopted iterative method for optimizing differentiable objective functions. In this paper, we propose and discuss a novel approach to scale up SGD in applications involving non-convex functions and large datasets. We address the bottleneck problem arising when using both shared and distributed memory. Typically, the former is bounded by limited computation resources and bandwidth whereas the latter suffers from communication overheads. We propose a unified distributed and parallel implementation of SGD (named DPSGD) that relies on both asynchronous distribution and lock-free parallelism. By combining two strategies into a unified framework, DPSGD is able to strike a better trade-off between local computation and communication. The convergence properties of DPSGD are studied for non-convex problems such as those arising in statistical modelling and machine learning. Our theoretical analysis shows that DPSGD leads to speed-up with respect to the number of cores and number of workers while guaranteeing an asymptotic convergence rate of O(1/root T) given that the number of cores is bounded by T-1/4 and the number of workers is bounded by T-1/2 where T is the number of iterations. The potential gains that can be achieved by DPSGD are demonstrated empirically on a stochastic variational inference problem (Latent Dirichlet Allocation) and on a deep reinforcement learning (DRL) problem (advantage actor critic - A2C) resulting in two algorithms: DPSVI and HSA2C. Empirical results validate our theoretical findings. Comparative studies are conducted to show the performance of the proposed DPSGD against the state-of-the-art DRL algorithms.
A microarray is a revolutionary tool that generates vast volumes of data that describe the expression profiles of genes under investigation that can be qualified as Big Data. Hadoop and Spark are efficient frameworks,...
详细信息
A microarray is a revolutionary tool that generates vast volumes of data that describe the expression profiles of genes under investigation that can be qualified as Big Data. Hadoop and Spark are efficient frameworks, developed to store and analyze Big Data. Analyzing microarray data helps researchers to identify correlated genes. Clustering has been successfully applied to analyze microarray data by grouping genes with similar expression profiles into clusters. The complex nature of microarray data obligated clustering methods to employ multiple evaluation functions to ensure obtaining solutions with high quality. This transformed the clustering problem into a Multi-Objective Problem (MOP). A new and efficient hybrid Multi-Objective Whale Optimization Algorithm with Tabu Search (MOWOATS) was proposed to solve MOPs. In this article, MOWOATS is proposed to analyze massive microarray datasets. Three evaluation functions have been developed to ensure an effective assessment of solutions. MOWOATS has been adapted to run in parallel using Spark over Hadoop computing clusters. The quality of the generated solutions was evaluated based on different indices, such as Silhouette and Davies-Bouldin indices. The obtained clusters were very similar to the original classes. Regarding the scalability, the running time was inversely proportional to the number of computing nodes.
In this paper, we propose distributed optimization methods to solve systems of linear equations. We provide convergence analysis for both continuous and discrete time computation models based on linear systems theory....
详细信息
In this paper, we propose distributed optimization methods to solve systems of linear equations. We provide convergence analysis for both continuous and discrete time computation models based on linear systems theory. It is shown that the proposed computation approaches work for very general linear equations, scalable with data sets and can be implemented in distributed or parallel fashion. Furthermore, we show that the discrete time algorithm admits constant update step size in the presence of additive uncertainties. This robustness feature makes the approach computationally efficient and supplementary to the existing approaches to deal with uncertainties such as stochastic (sub-)gradient methods and sample averaging.
Utilizing renewable energy sources to reduce carbon emission and minimizing the fuel cost for energy saving in the OPF (optimal power flow) problem will contribute to reducing the global warming effect from the power ...
详细信息
Utilizing renewable energy sources to reduce carbon emission and minimizing the fuel cost for energy saving in the OPF (optimal power flow) problem will contribute to reducing the global warming effect from the power generation sector. In this paper, we propose a DPOPF (distributed and parallel OPF) algorithm for the smart grid transmission system with renewable energy sources to account for the fast variation of the power generated by renewable energy sources. The proposed DPOPF algorithm is a combination of the recursive quadratic programming method and the Lagrange projected gradient method;it can achieve the complete decomposition and can be executed in the smart grid transmission system to make distributed and parallel computation possible. We also propose Petri nets to control the computational synchronization of the DPOPF algorithm under the asynchronous data arrival in the smart grid transmission system. The proposed DPOPF algorithm is applied to solve OPF problems in a smart grid transmission system with renewable energy sources on a 26-bus test system. The test results demonstrate the computational efficiency of the proposed DPOPF algorithm, which is fast enough to cope with the fast variation of the power generated by renewable energy sources, and justify the accuracy of the obtained solutions. (C) 2013 Elsevier Ltd. All rights reserved.
The block Fourier decomposition method recently proposed by the first author is a special method for decoupling any block tridiagonal matrix of the form K = block-tridiag [B, A, B], where A and B are square submatrice...
详细信息
ISBN:
(纸本)9783540405238
The block Fourier decomposition method recently proposed by the first author is a special method for decoupling any block tridiagonal matrix of the form K = block-tridiag [B, A, B], where A and B are square submatrices, into diagonal blocks. Unlike the traditional fast Poisson solver, block cyclic reductions, or the FACR algorithm, this approach does not require A and B be symmetric or commute. Presented in this paper is a parallel solver using this block decomposition method to solve linear systems whose coefficient matrices are of the form of K. We describe the computational procedure and implementation for parallel executions on distributed workstations. The performance from our numerical experiments is reported to demonstrate the usefulness of this approach.
暂无评论