The size of Transformer models is growing at an unprecedented rate. It has taken less than one year to reach trillion-level parameters since the release of GPT-3 (175B). Training such models requires both substantial ...
详细信息
ISBN:
(纸本)9781713845065
The size of Transformer models is growing at an unprecedented rate. It has taken less than one year to reach trillion-level parameters since the release of GPT-3 (175B). Training such models requires both substantial engineering efforts and enormous computing resources, which are luxuries most research teams cannot afford. In this paper, we propose PipeTransformer, which leverages automated elastic pipelining for efficient distributed training of Transformer models. In PipeTransformer, we design an adaptive on the fly freeze algorithm that can identify and freeze some layers gradually during training, and an elastic pipelining system that can dynamically allocate resources to train the remaining active layers. More specifically, PipeTransformer automatically excludes frozen layers from the pipeline, packs active layers into fewer GPUs, and forks more replicas to increase data-parallel width. We evaluate PipeTransformer using Vision Transformer (ViT) on ImageNet and BERT on SQuAD and GLUE datasets. Our results show that compared to the state-of-the-art baseline, PipeTransformer attains up to 2.83fold speedup without losing accuracy. We also provide various performance analyses for a more comprehensive understanding of our algorithmic and system-wise design. Finally, we have modularized our training system with flexible APIs and made the source code publicly available at https://***.
The power system constitutes a complex network where security is the priority. Contingency analysis emerges as a longstanding and widely embraced technique aimed at preemptively assessing the severity of potential out...
详细信息
ISBN:
(数字)9798350354379
ISBN:
(纸本)9798350354386
The power system constitutes a complex network where security is the priority. Contingency analysis emerges as a longstanding and widely embraced technique aimed at preemptively assessing the severity of potential outages and their ramifications, thereby facilitating necessary actions to uphold the power grid's security. Given the considerable number of components within the system, scrutinizing individual contingencies proves to be a laborious endeavour. However, it is imperative to identify those contingencies that could potentially trigger severe conditions within the power system. This is accomplished by using the fast decoupling load flow (FDLF) approach to compute performance indices, such as the reactive power performance index (PIv) and active performance index (PIp), for each transmission line outage. Subsequently, contingency ranking is performed within the MATLAB environment, furnishing operators with insights ranging from the most critical to the least significant outages in the power system. The integration of distributed generation (DG) into distribution networks is pursued with the objective of augmenting system performance indices. DGs are incorporated into load flow analysis employing either a constant power factor model (CPF) or a variable reactive power model (VRP), contingent upon the output power characteristics. These models are seamlessly integrated into the load flow analysis. The efficacy of this approach has been validated through testing on both IEEE 5-Bus and IEEE 14-Bus systems by integrating mathematical models of DGs.
AvesTerra is a distributed knowledge representation framework for integrating many large and disparate data systems and analytic components at global scale. This framework allows data created or curated by many differ...
详细信息
ISBN:
(纸本)9781450384049
AvesTerra is a distributed knowledge representation framework for integrating many large and disparate data systems and analytic components at global scale. This framework allows data created or curated by many different institutions to be linked into a single unified, dynamic knowledge representation structure. The resulting fabric provides participants with a means to engage in multidisciplinary research and collaboration spanning many information systems without requiring a sophisticated computer science understanding of the mechanics of "Big Data" manipulation. Furthermore, AvesTerra enables this integration without the need for centralized data aggregation or local high-performance computational infrastructure, leveraging instead the distributed resources of a diverse and highly distributed analytic *** a core technical level, AvesTerra consists of a system of peer-to-peer servers that collectively form a readily scalable knowledge space. The mathematical structure of this space is that of a generalized, recursive hypergraph, enabling the representation of complex dependency structures often encountered when working towards global scale. The framework incorporates numerous computational constructs including event publication and subscription, parallel threading and timer support, a unique distributed rendezvous mechanism for agent-based organization, privacy isolation, and semantic structure execution. This presentation provides an overview of the full framework and a sampling of the applications currently under development.
At present, the research of distributed power grid has made breakthroughs, and the proportion of power production is also increasing. The wide application of distributedgrid will have a great impact on the traditiona...
详细信息
In the traditional droop control, the VSG algorithm and the voltage compensation link are added to make the inverter have the large inertia characteristic of the synchronous generator (SG). The output voltage of each ...
详细信息
This paper proposes the use of SpaceWire and SpaceFibre to jointly build a high-speed and low-speed hybrid satellite-based integrated information transmission network, and an open and distributed high-performance comp...
This paper proposes the use of SpaceWire and SpaceFibre to jointly build a high-speed and low-speed hybrid satellite-based integrated information transmission network, and an open and distributed high-performance computing platform is designed and implemented. The built high performance computing platform is designed based on SpaceVPX architecture, so that SpaceWire and SpaceFbire serve as both the network for information transmission between the entire satellite equipment and equipment, and as a ”virtual backplane” to realize the information transmission between the hardware modules within the high-performance computing platform. Through the mutual cooperation and seamless connection between SpaceWire and SpaceFbire, the granularity of parallelcomputing of the on-board information system is refined from equipment to hardware modules, while with the design ideas of software-defined network and software-defined hardware, the hardware modules in physically different locations of the equipment are further formed into a logically integrated global information processing cluster to achieve the goal of multi-use and parallel reuse of the high performance computing platform with computing tasks assigned on demand and hardware modules enabled on demand. The goal is to lay the foundation for deep integration and linkage between satellite platforms and payload devices to meet the autonomous information processing and intelligence generation of satellites in orbit.
With the wide application of distributed generation system and the request of users to power supply quality and reliability of power supply, a new grid structure appeared in the society. Micro grid has solved the prob...
详细信息
ISBN:
(纸本)9783030311292;9783030311285
With the wide application of distributed generation system and the request of users to power supply quality and reliability of power supply, a new grid structure appeared in the society. Micro grid has solved the problem of large-scale access of distributed power supply, giving full play to all kinds of advantages of distributed power supply, meanwhile it also has brought a variety of other benefits to the user. Micro network will radically change the traditional way to deal with load growth and has great potential to reduce the energy consumption, improve the power system reliability and flexibility. At the same time, the grid is an important network model of the intelligent power distribution system in the future because its operation control system is the key to ensure the safe, economic and reliable operation of power grid. This paper mainly studies the present research situation of the technique of micro power grid operation control and its shortage, laying the foundation for other researchers.
The Internet of Things builds a digital world where all assets including people, machines, human-machine interfaces (HMIs), processes, services, etc. are internetworked. Digital Twins provide 1-to-1 digital replica of...
详细信息
ISBN:
(纸本)9781665404242
The Internet of Things builds a digital world where all assets including people, machines, human-machine interfaces (HMIs), processes, services, etc. are internetworked. Digital Twins provide 1-to-1 digital replica of physical assets and enable their networking and their goal-directed and standardized communication. The combination of assets and their corresponding Digital Twins are the "things", the nodes of the IoT. With the rapid development of loT and its pertinent technologies, more and more things will be mutually interconnected over the Internet. Hence, the security issues that come with it are gradually becoming sharper and sharper. In this paper, we focus on the topic of the secured communication among things - or more precisely their Digital Twins - and propose an approach related to OAuth2, asymmetrically encryption and digital signing to secure the communication within the newly developed Smart Systems Service Infrastructure ((SI)-I-3).
This special issue contains extended versions of selected papers from the 5th international Symposium on Formal Approaches to parallel and distributed Systems (4PAD 2018). The symposium was held in Orleans, France on ...
详细信息
This special issue contains extended versions of selected papers from the 5th international Symposium on Formal Approaches to parallel and distributed Systems (4PAD 2018). The symposium was held in Orleans, France on July 19th, 2020 in conjunction with the 2018 internationalconference on High Performance computing & Simulation (HPCS 2018). (C) 2020 Elsevier Inc. All rights reserved.
This work introduces novel parallel methods for weighted longest common subsequence (WLCS) and its generalization, allsubstrings WLCS. Previous work developed efficient algorithms for these problems via Monge matrix m...
详细信息
ISBN:
(纸本)9783030856656;9783030856649
This work introduces novel parallel methods for weighted longest common subsequence (WLCS) and its generalization, allsubstrings WLCS. Previous work developed efficient algorithms for these problems via Monge matrix multiplication, which is a limiting factor for further improvement. Diverging from these approaches, we relax the algorithm's optimality guarantee in a controlled way, using a different, natural dynamic program which can be sketched and solved in a divideand-conquer manner that is efficient to parallelize. Additionally, to compute the base case of our algorithm, we develop a novel and efficient method for all-substrings WLCS inspired by previous work on unweighted all-substrings LCS, exploiting the typically small range of weights. Our method fits in most parallel models of computation, including the PRAM and the BSP model. To the best of our knowledge this is the fastest (1 -epsilon)-approximation algorithm for all-substrings WLCS and WLCS in BSP. Further, this is the asymptotically fastest parallel algorithm for weighted LCS as the number of processors increases.
暂无评论