We describe JetLag, a Python-based environment that provides access to a distributed, interactive, asynchronous many-task (AMT) computing framework called Phylanx. This environment encompasses the entire computing pro...
详细信息
ISBN:
(纸本)9780738110868
We describe JetLag, a Python-based environment that provides access to a distributed, interactive, asynchronous many-task (AMT) computing framework called Phylanx. This environment encompasses the entire computing process, from a Jupyter front-end for managing code and results to the collection and visualization of performance data. We use a Python decorator to access the abstract syntax tree of Python functions and transpile them into a set of C++ data structures which are then executed by the HPX runtime. The environment includes services for sending functions and their arguments to run as jobs on remote resources. A set of Docker and Singularity containers are used to simplify the setup of the JetLag environment. The JetLag system is suitable for a variety of array computational tasks, including machine learning and exploratory data analysis.
Today's scientific software in the power system domain highly relies on CPU-based multi-core clusters. However, the rise of heterogeneous computing requires that parallel applications be executed on various hardwa...
详细信息
ISBN:
(纸本)9798350315097
Today's scientific software in the power system domain highly relies on CPU-based multi-core clusters. However, the rise of heterogeneous computing requires that parallel applications be executed on various hardware from different vendors. Thus, efficient solutions are emerging using compiler directives, communication protocols, high-level libraries, and execution policies. Power system dynamic simulation (DS) is critical to predicting and identifying real-time physical stability constraints with inevitable system component failures. This paper explores a multiprocessing heterogeneous solution for parallel DS. We design a single-source code realized through Message Passing Interface (MPI) and array computing at a high level. The performance test of a 30-second simulation on the largest 36864-bus-12288-generator system indicates that the program run with pure CPU-based parallelism finishes the computation in 14.6 seconds on a personal computer, and the program's CPU+GPU acceleration (hybrid switch) reaches more speedup with 2.33 seconds on a cluster node. The proposed solution significantly improves computational performance with high portability and adaptiveness to various hardware environments.
Despite advancements in the areas of parallel and distributed computing, the complexity of programming on High Performance computing (HPC) resources has deterred many domain experts, especially in the areas of machine...
详细信息
ISBN:
(纸本)9781728101781
Despite advancements in the areas of parallel and distributed computing, the complexity of programming on High Performance computing (HPC) resources has deterred many domain experts, especially in the areas of machine learning and artificial intelligence (AI), from utilizing performance benefits of such systems. Researchers and scientists favor high-productivity languages to avoid the inconvenience of programming in low-level languages and costs of acquiring the necessary skills required for programming at this level. In recent years, Python, with the support of linear algebra libraries like NumPy, has gained popularity despite facing limitations which prevent this code from distributed runs. Here we present a solution which maintains both high level programming abstractions as well as parallel and distributed efficiency. Phylanx, is an asynchronous array processing toolkit which transforms Python and NumPy operations into code which can be executed in parallel on HPC resources by mapping Python and NumPy functions and variables into a dependency tree executed by HPX, a general purpose, parallel, task-based runtime system written in C++. Phylanx additionally provides introspection and visualization capabilities for debugging and performance analysis. We have tested the foundations of our approach by comparing our implementation of widely used machine learning algorithms to accepted NumPy standards.
暂无评论