检索结果-内蒙古大学图书馆

A Mirror descent-Based Algorithm for Corruption-Tolerant distributed gradient descent

IEEE TRANSACTIONS ON SIGNAL PROCESSING 2025年 73卷 827-842页

作者： Wang, Shuche Tan, Vincent Y. F. Natl Univ Singapore Inst Operat Res & Analyt Singapore 117602 Singapore Natl Univ Singapore Dept Math Dept Elect & Comp Engn Singapore 117602 Singapore Natl Univ Singapore Inst Operat Res & Analyt Singapore 117602 Singapore

distributed gradient descent algorithms have come to the fore in modern machine learning, especially in parallelizing the handling of large datasets that are distributed across several workers. However, scant attention has been paid to analyzing the behavior of distributed gradient descent algorithms in the presence of adversarial corruptions instead of random noise. In this paper, we formulate a novel problem in which adversarial corruptions are present in a distributed learning system. We show how to use ideas from (lazy) mirror descent to design a corruption-tolerant distributed optimization algorithm. Extensive convergence analysis for (strongly) convex loss functions is provided for different choices of the stepsize. We carefully optimize the stepsize schedule to accelerate the convergence of the algorithm, while at the same time amortizing the effect of the corruption over time. Experiments based on linear regression, support vector classification, and softmax classification on the MNIST dataset corroborate our theoretical findings.

关键词： Vectors Servers Noise measurement Signal processing algorithms Machine learning algorithms Convergence Uplink Mirrors Downlink Machine learning distributed gradient descent mirror descent adversarial corruptions noisy channels convergence rates

来源：评论

学校读者我要写书评

暂无评论

distributed gradient descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers

引用

IEEE TRANSACTIONS ON SIGNAL PROCESSING 2019年第22期67卷 5850-5864页

作者： Cao, Xinyang Lai, Lifeng Univ Calif Davis Dept Elect & Comp Engn Davis CA 95616 USA

Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms.

关键词： Signal processing algorithms Servers Sociology Statistics Radio frequency Convergence Optimization Byzantine attacker convergence distributed gradient descent

来源：评论

学校读者我要写书评

暂无评论

Improving the Convergence of distributed gradient descent via Inexact Average Consensus

引用

JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS 2020年第2期185卷 504-521页

作者： Du, Bin Zhou, Jiazhen Sun, Dengfeng Purdue Univ Sch Aeronaut & Astronaut W Lafayette IN 47906 USA

It is observed that the inexact convergence of the well-known distributed gradient descent algorithm can be caused by inaccuracy of consensus procedures. Motivated by this, to achieve the improved convergence, we ensure the sufficiently accurate consensus via approximate consensus steps. The accuracy is controlled by a predefined sequence of consensus error bounds. It is shown that one can achieve exact convergence when the sequence decays to zero;furthermore, a linear convergence rate can be obtained when the sequence decays to zero linearly. To implement the approximate consensus step with given error bounds, an inexact average consensus scheme is proposed in a distributed manner. Due to the flexibility of choices of both consensus error bounds and consensus schemes, the proposed framework offers the potential to balance the communication and computation in distributed optimization.

关键词： distributed optimization distributed gradient descent Inexact convergence Average consensus

来源：评论

学校读者我要写书评

暂无评论

Convergence Analysis of a Continuous-Time distributed gradient descent Algorithm

引用

IEEE CONTROL SYSTEMS LETTERS 2021年第4期5卷 1339-1344页

作者： Zhang, Mengyao Liu, Xinzhi Liu, Jun Univ Waterloo Dept Appl Math Waterloo ON N2L 3G1 Canada

There are various methods designed for solving the distributed optimization problem with only local computation and communication. This letter discusses a continuous-time and distributed version of the gradient descent method for solving the distributed optimization problem. We prove that the convergence rate of this method matches those of the centralized gradient descent method and distributed consensus.

关键词： gradient methods Linear programming Acceleration Optimization Convergence Multi-agent systems distributed optimization continuous-time optimization algorithms distributed gradient descent convergence analysis

来源：评论

学校读者我要写书评

暂无评论

Directed-distributed gradient descent 53

Directed-Distributed Gradient Descent

引用

53rd Annual IEEE Allerton Conference on Communication, Control, and Computing (Allerton)

作者： Xi, Chenguang Khan, Usman A. Tufts Univ Dept Elect & Comp Engn 161 Coll Ave Medford MA 02155 USA

ISBN: (纸本)9781509018246

distributed gradient descent (DGD) is a well established algorithm to solve the minimization of a sum of multi-agents' objective functions in the network, with the assumption that the network is undirected, i.e., requiring the weight matrices to be doubly-stochastic. In this paper, we present a distributed algorithm, called Directed-distributed gradient descent (D-DGD), to solve the same problem over directed graphs. In each iteration of D-DGD, we augment an additional variable at each agent to record the change in the state evolution. The algorithm simultaneously constructs a row-stochastic matrix and a column-stochastic matrix instead of only a doubly-stochastic matrix. The analysis shows that D-DGD converges at a rate of O (lnk/root k).

关键词： distributed optimization multi-agent networks directed graphs distributed gradient descent

来源：评论

学校读者我要写书评

暂无评论

Fast Convergence for Stochastic and distributed gradient descent in the Interpolation Limit 26

Fast Convergence for Stochastic and Distributed Gradient Des...

引用

European Signal Processing Conference (EUSIPCO)

作者： Mitra, Partha P. Cold Spring Harbor Lab POB 100 Cold Spring Harbor NY 11724 USA

ISBN: (纸本)9789082797015

Modern supervised learning techniques, particularly those using deep nets, involve fitting high dimensional labelled data sets with functions containing very large numbers of parameters. Much of this work is empirical. Interesting phenomena have been observed that require theoretical explanations;however the non-convexity of the loss functions complicates the analysis. Recently it has been proposed that the success of these techniques rests partly in the effectiveness of the simple stochastic gradient descent algorithm in the so called interpolation limit in which all labels are fit perfectly. This analysis is made possible since the SGD algorithm reduces to a stochastic linear system near the interpolating minimum of the loss function. Here we exploit this insight by presenting and analyzing a new distributed algorithm for gradient descent, also in the interpolating limit. The distributed SGD algorithm presented in the paper corresponds to gradient descent applied to a simple penalized distributed loss function, L(w(1);...;w(n)) = Sigma(i)l(i)(w(i)) + mu Sigma() vertical bar w(i) - w(j)vertical bar(2). Here each node holds only one sample, and its own parameter vector. The notation < i;j > denotes edges of a connected graph defining the communication links between nodes. It is shown that this distributed algorithm converges linearly (ie the error reduces exponentially with iteration number), with a rate 1 - n/n lambda(min)(H) < R < 1 where lambda(min)(H) is the smallest nonzero eigenvalue of the sample covariance or the Hessian H. In contrast with previous usage of similar penalty functions to enforce consensus between nodes, in the interpolating limit it is not required to take the penalty parameter to infinity for consensus to occur. The analysis further reinforces the utility of the interpolation limit in the theoretical treatment of modern machine learning algorithms.

关键词： Interpolating limit Overfitting Stochastic gradient descent distributed gradient descent

来源：评论

学校读者我要写书评

暂无评论

ROBUST distributed gradient descent WITH ARBITRARY NUMBER OF BYZANTINE ATTACKERS

ROBUST DISTRIBUTED GRADIENT DESCENT WITH ARBITRARY NUMBER OF...

引用

IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

作者： Cao, Xinyang Lai, Lifeng Univ Calif Davis Dept Elect & Comp Engn Davis CA 95616 USA

ISBN: (纸本)9781538646588

Due to the grow of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers.

关键词： Byzantine attacker convergence distributed gradient descent

来源：评论

学校读者我要写书评

暂无评论

Directed-distributed gradient descent

Directed-Distributed Gradient Descent

引用

Annual Allerton Conference on Communication, Control, and Computing

作者： Chenguang Xi Usman A. Khan Department of Electrical and Computer Engineering Tufts University 161 College Ave Medford MA 02155 United States

ISBN: (纸本)9781509018253

关键词： distributed optimization multi-agent networks directed graphs distributed gradient descent

来源：评论

学校读者我要写书评

暂无评论

Robust distributed gradient descent with Arbitrary Number of Byzantine Attackers

Robust Distributed Gradient Descent with Arbitrary Number of...

引用

IEEE International Conference on Acoustics, Speech and Signal Processing

作者： Xinyang Cao Lifeng Lai Dept. of Elec. and Comp. Engr. University of California Davis

ISBN: (纸本)9781538646595

关键词： Byzantine attacker convergence distributed gradient descent Converge Attackers descent computational power algorithms Personnel

来源：评论

学校读者我要写书评

暂无评论

distributed optimization method for economic dispatch of active distribution networks via momentum with historical information and forecast gradient

引用

ENERGY REPORTS 2023年 9卷 1134-1144页

作者： Li, Bo Zhao, Ruifeng Lu, Jiangang Xin, Kuo Huang, Jinhua Lin, Guanqiang Chen, Jinrong Pang, Xueyue Elect Power Dispatching & Control Ctr Guangdong Po Guangzhou 510000 Peoples R China China Southern Power Grid Co Ltd Power Dispatch & Control Ctr Guangzhou 510670 Guangdong Peoples R China Elect Power Res Inst Guangdong Power Grid Co Ltd Guangzhou 510082 Guangdong Peoples R China Huizhou Power Supply Bur Guangdong Power Grid Co L Huizhou 516000 Guangdong Peoples R China Foshan Power Supply Bur Guangdong Power Grid Co Lt Foshan 528000 Guangdong Peoples R China China Energy Engn Grp Guangdong Elect Power Design Guangzhou 510663 Guangdong Peoples R China

In the context of vigorously developing new energy sources, the economic dispatch(ED) of active distribution network (ADN) is essential. Due to the single fault affecting the global and high communication cost in centralized scheduling, we propose a distributed gradient method, namely the fast Nesterov accelerated gradient method(FNAGM), which can solve the economic dispatch problem(EDP) in ADN. The distributed architecture does not need to collect global information and only uses a sparse communication network to complete communication exchanges. It is the distributed architecture that can protect users' private information and reduce communication pressure. By constructing the acceleration matrix based on the upper limit of the second derivative, the convergence speed can be effectively improved while satisfying the equality constraints. Eventually, the FNAGM combined with the historical momentum information and the forecast gradient, which is simulated in the bi-layer model of ADN via MATLAB. The verification results show that the algorithm with the historical momentum information and the forecast gradient can complete the optimal scheduling of controllable distributed generator(DG). What is more, the convergence efficiency performance is greatly improved.& COPY;2023 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://***/licenses/by-nc-nd/4.0/).

关键词： distributed gradient descent Active distribution network Economic dispatch

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：