Large deep learning models have shown great potential for delivering exceptional results in various applications. However, the training process can be incredibly challenging due to the models' vast parameter sizes...
详细信息
ISBN:
(纸本)9783031477201;9783031477218
Large deep learning models have shown great potential for delivering exceptional results in various applications. However, the training process can be incredibly challenging due to the models' vast parameter sizes, often consisting of hundreds of billions of parameters. Common distributed training methods, such as data parallelism, tensor parallelism, and pipeline parallelism, demand significant data communication throughout the process, leading to prolonged wait times for some machines in physically distant distributed systems. To address this issue, we propose a novel solution called Hulk, which utilizes a modified graph neural network to optimize distributed computing systems. Hulk not only optimizes data communication efficiency between different countries or even different regions within the same city, but also provides optimal distributed deployment of models in parallel. For example, it can place certain layers on a machine in a specific region or pass specific parameters of a model to a machine in a particular location. By using Hulk in experiments, we were able to improve the time efficiency of training large deep learning models on distributed systems by more than 20%. Our open source collection of unlabeled data: https://***/DLYuanGod/Hulk.
Context: Component-based middleware, such as the Lightweight CORBA Component Model, is increasingly used to implement enterprise distributed real-time and embedded (DRE) systems. In addition to supporting the quality-...
详细信息
Context: Component-based middleware, such as the Lightweight CORBA Component Model, is increasingly used to implement enterprise distributed real-time and embedded (DRE) systems. In addition to supporting the quality-of-service (QoS) requirements of individual DRE systems, component technologies must also support bounded latencies when effecting deployment changes to DRE systems in response to changing environmental conditions and operational requirements. Objective: The goals of this paper are to (1) study sources of inefficiencies and non-deterministic performance in deployment capabilities for DRE systems and (2) devise solutions to overcome these performance problems. Method: The paper makes two contributions to the study of the deployment and configuration of distributed component based applications. First, we analyze how conventional implementations of the OMG's deployment and Configuration (D&C) specification for component-based systems can significantly degrade deployment latencies. Second, we describe architectural changes and performance optimizations implemented within the Locality-Enhanced deployment and Configuration Engine (LE-DAnCE) implementation of the D&C specification to obtain efficient and deterministic deployment latencies. Results: We analyze the performance of LE-DAnCE in the context of component deployments on 10 nodes for a representative DRE system consisting of 1000 components and in a cluster environment with up to 100 nodes. Our results show LE-DAnCE's optimizations provide a bounded deployment latency of less than 2 s for the 1000 component scenario with just a 4 percent jitter. Conclusion: The improvements contained in the LE-DAnCE infrastructure provide an efficient and scaleable standards-based deployment system for component-based enterprise DRE systems. In particular, deployment time parallelism can improve deployment latency significantly, both during pre-deployment analysis of the deployment plan and during the process of instal
暂无评论