检索结果-内蒙古大学图书馆

Model averaging in distributed machine learning: a case study with Apache Spark

VLDB JOURNAL 2021年第4期30卷 693-712页

作者： Guo, Yunyan Zhang, Zhipeng Jiang, Jiawei Wu, Wentao Zhang, Ce Cui, Bin Li, Jianzhong Harbin Inst Technol Mass Data Comp Res Ctr Harbin 150001 Peoples R China Peking Univ Sch EECS Beijing 100871 Peoples R China Microsoft Res Redmond WA USA Swiss Fed Inst Technol Dept Comp Sci CH-8092 Zurich Switzerland

The increasing popularity of Apache Spark has attracted many users to put their data into its ecosystem. On the other hand, it has been witnessed in the literature that Spark is slow when it comes to distributed machine learning (ML). One resort is to switch to specialized systems such as parameter servers, which are claimed to have better performance. Nonetheless, users have to undergo the painful procedure of moving data into and out of Spark. In this paper, we investigate performance bottlenecks of MLlib (an official Spark package for ML) in detail, by focusing on analyzing its implementation of stochastic gradient descent (SGD)-the workhorse under the training of many ML models. We show that the performance inferiority of Spark is caused by implementation issues rather than fundamental flaws of the bulk synchronous parallel (BSP) model that governs Spark's execution: we can significantly improve Spark's performance by leveraging the well-known "model averaging" (MA) technique in distributed ML. Indeed, model averaging is not limited to SGD, and we further showcase an application of MA to training latent Dirichlet allocation (LDA) models within Spark. Our implementation is not intrusive and requires light development effort. Experimental evaluation results reveal that the MA-based versions of SGD and LDA can be orders of magnitude faster compared to their counterparts without using MA.

关键词： distributed machine learning Apache Spark MLlib Generalized linear models Latent Dirichlet allocation

来源：评论

学校读者我要写书评

暂无评论

DPS: Dynamic Pricing and Scheduling for distributed machine learning Jobs in Edge-Cloud Networks

引用

IEEE TRANSACTIONS ON MOBILE COMPUTING 2023年第11期22卷 6377-6393页

作者： Zhou, Ruiting Wang, Ne Huang, Yifeng Pang, Jinlong Chen, Hao Southeast Univ Sch Comp Sci & Engn Nanjing 211189 Peoples R China Wuhan Univ Sch Comp Sci Wuhan 430072 Peoples R China Wuhan Univ Sch Cyber Sci & Engn Key Lab Aerosp Informat Secur & Trusted Comp Minist Educ Wuhan 430072 Peoples R China Huawei Nanjing 210019 Peoples R China

5G and Internet of Things stimulate smart applications of edge computing, such as autonomous driving and smart city. As edge computing power increases, more and more machine learning (ML) jobs will be trained in the edge-cloud network, adopting the parameter server (PS) architecture. Due to the distinct features of the edge (low-latency and the scarcity of resources), the cloud (high delay and rich computing capacity) and ML jobs (frequent communication between workers and PSs and unfixed runtime), existing cloud job pricing and scheduling algorithms are not applicable. Therefore, how to price, deploy and schedule ML jobs in the edge-cloud network becomes a challenging problem. To solve it, we propose an auction-based online framework DPS. DPS consists of three major parts: job admission control, price function design and scheduling orchestrator. DPS dynamically prices workers and PSs based on historical job information and real-time system status, and decides whether to accept the job according to the deployment cost. DPS then deploys and schedules accepted ML jobs to pursue the maximum social welfare. Through theoretical analysis, we prove that DPS can achieve a good competition ratio and truthfulness in polynomial time. Large-scale simulations and testbed experiments show that DPS can improve social welfare by at least 95%, compared with benchmark algorithms in today's cloud system.

关键词： distributed machine learning online pricing online scheduling parameter server architecture edge-cloud networks

来源：评论

学校读者我要写书评

暂无评论

UbiNN: A Communication Efficient Framework for distributed machine learning in Edge Computing

引用

IEEE TRANSACTIONS ON NETWORK SCIENCE AND ENGINEERING 2023年第6期10卷 3368-3383页

作者： Li, Ke Chen, Kexun Luo, Shouxi Zhang, Honghao Fan, Pingzhi Southwest Jiaotong Univ Sch Comp & Artificial Intelligence Chengdu 611756 Peoples R China Southwest Jiaotong Univ Prov Key Lab Informat Coding &Transmiss Chengdu 611756 Peoples R China

Deployment of distributed machine learning at the edge is conducive to reducing latency and protecting privacy associated with transmitting data back to the cloud. Nonetheless, as machine learning models scale, bandwidth resources are limited and heterogeneity in data collection from heterogeneous edge devices is visible, resulting in low model accuracy and high communication overhead. In this paper, a novel distributed deep learning framework called ubiquitous neural network (UbiNN) is proposed to improve communication efficiency without affecting the accuracy of the local neural network model at the edge or the global neural network model in the cloud. As for the accuracy of the neural network model, a common dataset with a small portion of insensitive data is constructed for training the neural network model, and its accuracy is enhanced by a new algorithm based on knowledge distillation and covariance computation (KDCC). Experimental results demonstrate that the test accuracy of UbiNN is extremely close to the centralized machine learning and non-federated learning scheme DDNN and up to 18.74% better than that of other classic federated learning schemes when using public datasets such as MNIST, CIFAR-10, and REUTERS-21578. Meanwhile, communication overheads are substantially reduced in terms of data transmission volume and latency.

关键词： Communication-efficient distributed machine learning edge computing knowledge distillation UbiNN

来源：评论

学校读者我要写书评

暂无评论

Toward Efficient Online Scheduling for distributed machine learning Systems

引用

IEEE TRANSACTIONS ON NETWORK SCIENCE AND ENGINEERING 2022年第4期9卷 1951-1969页

作者： Yu, Menglu Liu, Jia Wu, Chuan Ji, Bo Bentley, Elizabeth S. Iowa State Univ Dept Comp Sci Ames IA 50011 USA Ohio State Univ Dept Elect & Comp Engn Columbus OH 43210 USA Virginia Tech Dept Comp Sci Blacksburg VA 24061 USA Univ Hong Kong Dept Comp Sci Hong Kong 999077 Peoples R China Air Force Res Lab Informat Directorate Rome NY 13441 USA

Recent years have witnessed a rapid growth of distributed machine learning (ML) frameworks, which exploit the massive parallelism of computing clusters to expedite ML training. However, the proliferation of distributed ML frameworks also introduces many unique technical challenges in computing system design and optimization. In a networked computing cluster that supports a large number of training jobs, a key question is how to design efficient scheduling algorithms to allocate workers and parameter servers across different machines to minimize the overall training time. Toward this end, in this paper, we develop an online scheduling algorithm that jointly optimizes resource allocation and locality decisions. Our main contributions are three-fold: i) We develop a new analytical model that considers both resource allocation and locality;ii) Based on an equivalent reformulation and observations on the worker-parameter server locality configurations, we transform the problem into a mixed packing and covering integer program, which enables approximation algorithm design;iii) We propose a meticulously designed approximation algorithm based on randomized rounding and rigorously analyze its performance. Collectively, our results contribute to the state of the art of distributed ML system optimization and algorithm design.

关键词： Servers Training Optimization Scheduling algorithms Resource management Approximation algorithms Heuristic algorithms Online resource scheduling distributed machine learning approximation algorithm

来源：评论

学校读者我要写书评

暂无评论

Computation Scheduling for distributed machine learning With Straggling Workers

引用

IEEE TRANSACTIONS ON SIGNAL PROCESSING 2019年第24期67卷 6270-6284页

作者： Amiri, Mohammad Mohammadi Gunduz, Deniz Princeton Univ Dept Elect Engn Princeton NJ 08544 USA Imperial Coll London Dept Elect & Elect Engn London SW7 2AZ England

We study scheduling of computation tasks across $n$ workers in a large scale distributed learning problem with the help of a master. Computation and communication delays are assumed to be random, and redundant computations are assigned to workers in order to tolerate stragglers. We consider sequential computation of tasks assigned to a worker, while the result of each computation is sent to the master right after its completion. Each computation round, which can model an iteration of the stochastic gradient descent (SGD) algorithm, is completed once the master receives $k$ distinct computations, referred to as the computation target. Our goal is to characterize the average completion time as a function of the computation load, which denotes the portion of the dataset available at each worker, and the computation target. We propose two computation scheduling schemes that specify the tasks assigned to each worker, as well as their computation schedule, i.e., the order of execution. Assuming a general statistical model for computation and communication delays, we derive the average completion time of the proposed schemes. We also establish a lower bound on the minimum average completion time by assuming prior knowledge of the random delays. Experimental results carried out on Amazon EC2 cluster show a significant reduction in the average completion time over existing coded and uncoded computing schemes. It is also shown numerically that the gap between the proposed scheme and the lower bound is relatively small, confirming the efficiency of the proposed scheduling design.

关键词： Task analysis Delays Processor scheduling Computational modeling Encoding Schedules Decoding distributed machine learning uncoded computing computation scheduling straggling workers

来源：评论

学校读者我要写书评

暂无评论

Online Scheduling of distributed machine learning Jobs for Incentivizing Sharing in Multi-Tenant Systems

引用

IEEE TRANSACTIONS ON COMPUTERS 2023年第3期72卷 653-667页

作者： Wang, Ne Zhou, Ruiting Han, Ling Chen, Hao Li, Zongpeng Wuhan Univ Sch Comp Sci Wuhan 430072 Peoples R China Wuhan Univ Sch Comp Sci Sch Cyber Sci & EngnMinist Educ Key Lab Aerosp Informat Secur & Trusted Comp Wuhan 430072 Peoples R China Duke Univ Dept Comp Sci Durham NC 27708 USA Huawei Nanjing 211100 Jiangsu Peoples R China Tsinghua Univ Inst Network Sci & Cyberspace Beijing 100190 Peoples R China

To save cost, companies usually train machine learning (ML) models on a shared multi-tenant system. In this cooperative environment, one of the fundamental challenges is how to distribute resources fairly among tenants such that each tenant is satisfied. A satisfactory allocation policy needs to meet the following properties. First, the performance of each tenant in the shared cluster is at least the same as that in its exclusive cluster partition. Second, no tenant can get more benefits by lying about its demands. Third, tenants cannot use the idle resources of others for free. Moreover, the resource allocation for ML workloads should avoid costly migration overhead. To this end, we propose a three-layer scheduling framework Astraea: i) a batch scheduling framework groups unprocessed ML jobs into multiple batches;ii) a round-by-round algorithm enables tenants to reserve their share of resources and schedule ML jobs in a non-preemptive manner;iii) one-round algorithm based on the primal-dual approach and posted pricing framework, which encourages tenants to report truthful demands and computes a schedule with maximal revenue for each ML job. Besides, our algorithm also allows tenants to trade unused resources on a paid basis. Astraea is proven to achieve performance guarantee, polynomial time complexity and some desirable properties of resource sharing, including batch sharing incentive, strategy-proofness, Pareto efficiency and gain-as-you-contribute fairness. Extensive trace-driven simulations show that our algorithm advances in both fairness and cluster efficiency by at least 20% compared to three state-of-the-art baselines.

关键词： distributed machine learning fairness multi-tenant system online scheduling parameter server architecture

来源：评论

学校读者我要写书评

暂无评论

When In-Network Computing Meets distributed machine learning

引用

IEEE NETWORK 2024年第5期38卷 238-246页

作者： Zhu, Haowen Jiang, Wenchao Hong, Qi Guo, Zehua Beijing Inst Technol Beijing 100081 Peoples R China Beijing Inst Technol Zhengzhou Res Inst Zhengzhou 450000 Peoples R China Singapore Univ Technol & Design Informat Syst Technol & Design Singapore 487372 Singapore

Emerging In-Network Computing (INC) technique provides a new opportunity to improve application's performance by using network programmability, computational capability, and storage capacity enabled by programmable switches. One typical application is distributed machine learning (DML), which accelerates machine learning training by employing multiple works to train model parallelly. This paper introduces INC-based DML systems, analyzes performance improvement from using INC, and overviews current studies of INC-based DML systems. We also propose potential research directions for applying INC to DML systems.

关键词： Training Computational modeling Servers Data models machine learning Synchronization Performance evaluation distributed machine learning In-Network Computing machine learning Programmable Switch

来源：评论

学校读者我要写书评

暂无评论

GreedW: A Flexible and Efficient Decentralized Framework for distributed machine learning

引用

IEEE TRANSACTIONS ON COMPUTERS 2024年第3期73卷 801-814页

作者： Wang, Ting Jiang, Xin Li, Qin Cai, Haibin East China Normal Univ Software Engn Inst Engn Res Ctr Software Hardware Codesign Technol & Shanghai Key Lab Trustworthy Comp Shanghai 200062 Peoples R China

With the ever-increasing demand for computing power in deep learning, distributed training techniques have proven to be effective in meeting these demands. However, current existing state-of-the-art distributed training frameworks, such as Parameter Server (PS), Ring-All-Reduce, and their varieties, still face significant challenges. In particular, the existence of communication bottlenecks can severely limit the efficiency and scalability of distributed training frameworks, making it difficult to fully and effectively exert the computing power of large-scale clusters, especially in the presence of dynamic and ever-changing network environments. To address these issues and further maximize the utilization of the computing power of clusters, in this paper we propose an efficient and dynamic distributed training framework, named GreedW. GreedW can greatly improve the training efficiency of workers by dynamically constructing an adaptive customized communication network and adaptively scheduling the workload. Specifically, GreedW employs a greedy strategy to dynamically construct the communication network tree in each iteration for gradient transmission with minimum communication cost and applies a heterogeneity-aware workload allocation scheme to adaptively balance the heavy traffic across heterogeneous workers in the cluster taking into account the available computing capabilities of each node, which effectively alleviates the network bottleneck. It is worth noting that GreedW is enabled to dynamically adjust the assigned job on each worker node based on their completion time during each round of model aggregation to ensure that each worker node completes its assignments around the same time, thus mitigating the intractable straggler issue and minimizing their idle waiting time. Comprehensive experimental evaluations on three different-scaled training models (i.e., Mnist-2NN, Mnist-CNN, and TextCNN) for image recognition and natural language processing tasks demonstr

关键词： Training Costs Computational modeling Task analysis Servers Adaptation models Pipelines distributed machine learning heterogeneous cluster workload allocation

来源：评论

学校读者我要写书评

暂无评论

A scalable distributed machine learning approach for attack detection in edge computing environments

引用

JOURNAL OF PARALLEL AND distributed COMPUTING 2018年 119卷 18-26页

作者： Kozik, Rafal Choras, Michal Ficco, Massimo Palmieri, Francesco UTP Univ Sci & Technol Bydgoszcz Inst Telecommun & Comp Sci Bydgoszcz Poland Univ Campania Luigi Vanvitelli Dept Ind & Informat Engn Aversa CE Italy Univ Salerno Dept Comp Sci Fisciano SA Italy

The ever-increasing number of loT applications and cyber-physical services is introducing significant challenges associated to their cyber-security. Due to the constrained nature of the involved devices, some heavier computational tasks, such as deep traffic inspection and classification, essential for implementing automatic attack detection systems, are moved on specialized "edge" devices, in order to distribute the processing intelligence near to the data sources. These edge devices are mainly capable of effectively running pre-built classification models but have not enough storage and processing capabilities to build and upgrade such models from huge volumes of field training data, imposing a serious barrier to the deployment of such solutions. This work leverages the flexibility of cloud-based architectures, together with the recent advancements in the area of large-scale machine learning for shifting the more computationally-expensive and storage-demanding operations to the cloud in order to benefit of edge computing capabilities only for effectively performing traffic classification based on sophisticated Extreme learning machines models that are pre-built over the cloud. (C) 2018 Elsevier Inc. All rights reserved.

关键词： loT Edge computing Extreme learning machines distributed machine learning Attack detection

来源：评论

学校读者我要写书评

暂无评论

Edge/Cloud Infinite-Time Horizon Resource Allocation for distributed machine learning and General Tasks

引用

IEEE TRANSACTIONS ON NETWORK AND SERVICE MANAGEMENT 2024年第1期21卷 697-713页

作者： Sartzetakis, Ippokratis Soumplis, Polyzois Pantazopoulos, Panagiotis Katsaros, Konstantinos V. Sourlas, Vasilis Varvarigos, Emmanouel Natl Tech Univ Athens Sch Elect & Comp Engn Athens 15773 Greece Natl Tech Univ Athens Inst Commun & Comp Syst Athens 15773 Greece

Edge computing has emerged as a computing paradigm where the application and data processing takes place close to the end devices. It decreases the distances over which data transfers are made, offering reduced delay and fast speed of action for general data processing and store/retrieve jobs. The benefits of edge computing can also be reaped for distributed computation algorithms, where the cloud also plays an assistive role. In this context, an important challenge is to allocate the required resources at both edge and cloud to carry out the processing of data that are generated over a continuous ("infinite") time horizon. This is a complex problem due to the variety of requirements (resource needs, accuracy, delay, etc.) that may be posed by each computation algorithm, as well as the heterogeneous resources' features (e.g., processing, bandwidth). In this work, we develop a solution for serving weakly coupled general distributed algorithms, with emphasis on machine learning algorithms, at the edge and/or the cloud. We present a dual-objective Integer Linear Programming formulation that optimizes monetary cost and computation accuracy. We also introduce efficient heuristics to perform the resource allocation. We examine various distributed ML allocation scenarios using realistic parameters from actual vendors. We quantify trade-offs related to accuracy, performance and cost of edge/cloud bandwidth and processing resources. Our results indicate that among the many parameters of interest, the processing costs seem to play the most important role for the allocation decisions. Finally, we explore interesting interactions between target accuracy, monetary cost and delay.

关键词： Cloud and edge computing distributed computing distributed machine learning inference training resource allocation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：