检索结果-内蒙古大学图书馆

distributed optimization for deep learning with gossip exchange

NEUROCOMPUTING 2019年 330卷 287-296页

作者： Blot, Michael Picard, David Thome, Nicolas Cord, Matthieu Sorbonne Univ CNRS UPMC Univ Paris 06 LIP6 UMR 7606 4 Pl Jussieu F-75005 Paris France Univ Cergy Pontoise CNRS Univ Paris Seine ENSEAETIS UMR 8051 Paris France Conservatoire Natl Arts & Metiers CEDRIC 292 Rue St Martin F-75003 Paris France

We address the issue of speeding up the training of convolutional neural networks by studying a distributed method adapted to stochastic gradient descent. Our parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way of sharing information between different threads based on gossip algorithms that show good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. (C) 2018 Published by Elsevier B.V.

关键词： Optimization distributed gradient descent Gossip Deep Learning Neural networks

来源：评论

学校读者我要写书评

暂无评论

An Approximate distributed gradient Estimation Method for Networked System Optimization Under Limited Communications

引用

IEEE TRANSACTIONS ON SYSTEMS MAN CYBERNETICS-SYSTEMS 2020年第12期50卷 5142-5151页

作者： Wang, Jing Pham, Khanh D. Bradley Univ Dept Elect & Comp Engn Peoria IL 61625 USA Air Force Res Lab AFRL RVSWS Kirtland AFB Albuquerque NM 87117 USA

This paper considers the networked system optimization problem by cooperatively finding an approximately optimal solution to the overall network convex cost function, which is the sum of the individual cost functions of nodes (agents) in a networked system. A new distributed gradient descent algorithm is proposed based on the distributed estimation of the sum of gradients of individual cost functions using a consensus-type coordination algorithm. The proposed algorithm can address unknown directed communication topologies and only requires limited communications and information exchanges among nodes in the networked system. Under the assumption that the communication graph is strongly connected, the convergence of the proposed algorithm is rigorously analyzed. Simulation examples are presented to demonstrate the applicability and effectiveness of the proposed algorithm.

关键词： Cost function Estimation Topology Approximation algorithms Computational modeling Resource management Consensus distributed gradient descent limited communication networked system optimization

来源：评论

学校读者我要写书评

暂无评论

Scalable Evidential K-Nearest Neighbor Classification on Big Data

引用

IEEE TRANSACTIONS ON BIG DATA 2024年第3期10卷 226-237页

作者： Gong, Chaoyu Demmel, Jim You, Yang Natl Univ Singapore Sch Comp Singapore 119077 Singapore Univ Calif Berkeley Dept Elect Engn & Comp Sci Berkeley CA 94720 USA

The K -Nearest Neighbor (K-NN) algorithm has garnered widespread utilization in real-world scenarios, due to its exceptional interpretability that other classification algorithms may not have. The evidential K-NN (EK-NN) algorithm builds upon the same nearest neighbor search procedure as K-NN, and provides more informative classification outcomes. However, EK-NN is not practical for Big Data because it is computationally complex. First, the search for K nearest neighbors of test samples from n training samples requires O(n(2)) operations. Additionally, estimating parameters involves performing complicated matrix calculations that increase in scale as the dataset becomes larger. To address these issues, we propose two scalable EK-NN classifiers, Global Exact EK-NN and Local Approximate EK-NN, under the distributed Spark framework. Along with the Local Approximate EK-NN, a new distributed gradient descent algorithm is developed to learn parameters. Data parallelism is used to reduce negative impacts caused by data distribution differences. Experimental results show that Our algorithms are able to achieve state-of-the-art scaling efficiency and accuracy on large datasets with more than 10 million samples.

关键词： Evidential K-nearest neighbor classification Big Data spark distributed gradient descent

来源：评论

学校读者我要写书评

暂无评论

D-DistADMM: A O(1/k) distributed ADMM for distributed Optimization in Directed Graph Topologies 59

D-DistADMM: A O(1/k) Distributed ADMM for Distributed Optimi...

引用

59th IEEE Conference on Decision and Control (CDC)

作者： Khatana, Vivek Salapaka, Murti, V Univ Minnesota Dept Elect & Comp Engn Minneapolis MN 55455 USA

ISBN: (纸本)9781728174471

We focus on the problem of minimizing a finite sum f(x) = Sigma(n)(i=1) f(i)(x) of n functions f(i), where f(i) are convex and available only locally to an agent i. The n agents are connected in a directed network G(V,E). In this article, we present the Directed-distributed Alternating Direction Method of Multiplier (D-DistADMM) Algorithm, which is an Alternating Direction Method of Multiplier (ADMM) based scheme and utilizes a finite-time "approximate" consensus method to solve the above optimization problem distributively. At each iteration of the proposed scheme the agents solve their local optimization problem and utilize an approximate consensus protocol to update a local estimate of the global optimization variable. We show that for convex and notnecessarily differentiable objective functions the proposed D-DistADMM method converges at a rate O(1/k), where k is the iteration counter, in terms the difference between the Lagrangian function evaluated at any iteration k of the D-DistADMM algorithm and the optimal solution. We further demonstrate the features of our algorithm by solving a distributed least squares problem.

关键词： distributed optimization distributed gradient descent multi-agent networks finite-time consensus

来源：评论

学校读者我要写书评

暂无评论

gradient-Consensus Method for distributed Optimization in Directed Multi-Agent Networks

Gradient-Consensus Method for Distributed Optimization in Di...

引用

American Control Conference (ACC)

作者： Khatana, Vivek Saraswat, Govind Patel, Sourav Salapaka, Murti, V Univ Minnesota Dept Elect & Comp Engn Minneapolis MN 55455 USA

ISBN: (纸本)9781538682661

In this article, a distributed optimization problem for minimizing a sum, Sigma(n)(i=1) f(i), of convex objective functions, f(i), on directed graph topologies is addressed. Here each function f(i) is a function of n variables, private to agent i which defines the agent's objective. These f(i)'s are assumed to be Lipschitz-differentiable convex functions. For solving this optimization problem, we develop a novel distributed algorithm, which we term as the gradient-consensus method. The gradient-consensus scheme uses a finite-time terminated consensus protocol called rho-consensus, which allows each local estimate to be rho-close to each other at every iteration. The parameter rho is a fixed constant independent of the network size and topology. It is shown that the estimate of the optimal solution at any local agent i converges geometrically to the optimal solution within an O(rho) neighborhood, where rho can be chosen to be arbitrarily small.

关键词： distributed optimization distributed gradient descent multi-agent networks finite-time consensus

来源：评论

学校读者我要写书评

暂无评论

distributed Data Minimization for Decentralized Collaborative Filtering Systems 23

Distributed Data Minimization for Decentralized Collaborativ...

引用

24th International Conference on distributed Computing and Networking (ICDCN)

作者： Eichinger, Tobias Kuepper, Axel Tech Univ Berlin Serv Ctr Networking Berlin Germany

ISBN: (纸本)9781450397964

Data minimization is a legal principle that mandates the limitation of personal data to a necessary minimum in order to protect the privacy of individuals. In this light, we address ourselves to decentralized collaborative filtering systems in which individual users interact with each other instead of institutional recommendation providers to obtain recommendations. Decentralized collaborative filtering systems, in which data is distributed over all users, bear a privacy benefit over traditional centralized systems in which data is kept at a single authoritative node. We propose the first formal definition of data minimization for use in distributed systems in order to combine the privacy benefits of data minimization and data distribution. On the basis of this definition, we propose a distributed data minimization scheme based on distributed gradient descent (DGD) that solves the data minimization problem for decentralized collaborative filtering systems. Here, data minimization represents a collection minimization problem in which users aim to collect necessary minimum amounts of rating data from other users in the system. We find by means of simulation that the overall amount of data collected by all users can be reduced significantly (> 14.6%) without jeopardizing recommendation performance, when users coordinate data collection. We thus prove that users collect unnecessarily much data, when they do not coordinate data collection.

关键词： data minimization recommender systems decentralized collaborative filtering distributed systems distributed gradient descent redundancy minimization

来源：评论

学校读者我要写书评

暂无评论

WEIGHTED gradient CODING WITH LEVERAGE SCORE SAMPLING

WEIGHTED GRADIENT CODING WITH LEVERAGE SCORE SAMPLING

引用

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

作者： Charalambides, Neophytos Pilanci, Mert Hero, Alfred O., III Univ Michigan EECS Dept Ann Arbor MI 48109 USA Stanford Univ EE Dept Stanford CA 94305 USA

ISBN: (纸本)9781509066315

A major hurdle in machine learning is scalability to massive datasets. Approaches to overcome this hurdle include compression of the data matrix and distributing the computations. Leverage score sampling provides a compressed approximation of a data matrix using an importance weighted subset. gradient coding has been recently proposed in distributed optimization to compute the gradient using multiple unreliable worker nodes. By designing coding matrices, gradient coded computations can be made resilient to stragglers, which are nodes in a distributed network that degrade system performance. We present a novel weighted leverage score approach, that achieves improved performance for distributed gradient coding by utilizing an importance sampling.

关键词： Reed-Solomon codes distributed gradient descent linear regression sketching straggler mitigation

来源：评论

学校读者我要写书评

暂无评论

Tree gradient Coding Considering Communication Delays and Partial Stragglers 59

Tree Gradient Coding Considering Communication Delays and Pa...

引用

59th Annual IEEE International Conference on Communications (IEEE ICC)

作者： Shah, Raj Tiwari, Utsav Thomas, Anoop Indian Inst Technol Bhubaneswar Sch Elect Sci Bhubaneswar India

ISBN: (纸本)9781728190549

There are two major problems while training large machine learning models using distributed gradient descent. The first is the problem of straggling workers, and the second is the communication delays in transmitting the computed gradient. In Tree gradient Coding (TGC) architecture, the workers are arranged in a tree topology, and the data partitions are redundantly assigned to these workers, providing us resilience to straggling workers. In TGC the effect of the communication delays and the partial straggling behavior of the workers are not considered while distributing the computation load. In this paper, an expression for computation load in TGC considering the communication delays and the partial stragglers is derived. Moreover, the proposed technique is implemented on cloud-based VMs and experimental results are obtained. A speedup of up to 23.95% is observed compared to the traditional TGC scheme.

关键词： Coding Theory distributed gradient descent gradient Coding Tree gradient Coding

来源：评论

学校读者我要写书评

暂无评论

On data minimization and anonymity in pervasive mobile-to-mobile recommender systems

引用

PERVASIVE AND MOBILE COMPUTING 2024年 103卷

作者： Eichinger, Tobias Kuepper, Axel Tech Univ Berlin Serv Centr Networking SNET Ernst Reuter Pl 7 D-10587 Berlin Germany

Data minimization is a legal principle that mandates limiting the collection of personal data to a necessary minimum. In this context, we address ourselves to pervasive mobile -to -mobile recommender systems in which users establish ad hoc wireless connections between their mobile computing devices in physical proximity to exchange ratings that represent personal data on which they calculate recommendations. The specific problem is: How can users minimize the collection of ratings over all users while only being able to communicate with a subset of other users in physical proximity? A main difficulty is the mobility of users, which prevents, for instance, the creation and use of an overlay network to coordinate data collection. Users, therefore, have to decide whether to exchange ratings and how many when an ad hoc wireless connection is established. We model the randomness of these connections and apply an algorithm based on distributed gradient descent to solve the distributed data minimization problem at hand. We show that the algorithm robustly produces the least amount of connections and also the least amount of collected ratings compared to an array of baselines. We find that this simultaneously reduces the chances of an attacker relating users to ratings. In this sense, the algorithm also preserves the anonymity of users, yet only of those users who do not establish an ad hoc wireless connection with each other. Users who do establish a connection with each other are trivially not anonymous toward each other. We find that users can further minimize data collection and preserve their anonymity if they aggregate multiple ratings on the same item into a single rating and change their identifiers between connections.

关键词： Data minimization Anonymity Decentralized recommender system distributed gradient descent Pervasive computing Mobile computing

来源：评论

学校读者我要写书评

暂无评论

gradient-Consensus Method for distributed Optimization in Directed Multi-Agent Networks

Gradient-Consensus Method for Distributed Optimization in Di...

引用

American Control Conference

作者： Vivek Khatana Govind Saraswat Sourav Patel Murti V. Salapaka department of Electrical and Computer Engineering University of Minnesota Minneapolis USA

ISBN: (数字)9781538682661

ISBN: (纸本)9781538682678

In this article, a distributed optimization problem for minimizing a sum, Σ_i = 1~n, f_i, of convex objective functions, f_i, on directed graph topologies is addressed. Here each function f i is a function of n variables, private to agent i which defines the agent's objective. These f i 's are assumed to be Lipschitz-differentiable convex functions. For solving this optimization problem, we develop a novel distributed algorithm, which we term as the gradient-consensus method. The gradient-consensus scheme uses a finite-time terminated consensus protocol called ρ-consensus, which allows each local estimate to be ρ-close to each other at every iteration. The parameter ρ is a fixed constant independent of the network size and topology. It is shown that the estimate of the optimal solution at any local agent i converges geometrically to the optimal solution within an O(ρ) neighborhood, where ρ can be chosen to be arbitrarily small.

关键词： distributed optimization distributed gradient descent multi-agent networks finite-time consensus NETWORKING AGENT optimal solution Mentoring distributed algorithms Distribute Convex function

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：