检索结果-内蒙古大学图书馆

Minimization of high computational cost in data preprocessing and modeling using mpi4py

MACHINE LEARNING WITH APPLICATIONS 2023年 13卷

作者： Oluwasakin, E. Torku, T. Tingting, S. Yinusa, A. Hamdan, S. Poudel, S. Hasan, N. Vargas, J. Poudel, K. Middle Tennessee State Univ Computat & Data Sci Program Murfreesboro TN USA Middle Tennessee State Univ Comp Sci Program Murfreesboro TN 37132 USA Middle Tennessee State Univ Engn Technol Murfreesboro TN USA

Data preprocessing is a fundamental stage in deep learning modeling and serves as the cornerstone of reliable data analytics. These deep learning models require significant amounts of training data to be effective, with small datasets often resulting in overfitting and poor performance on large datasets. One solution to this problem is parallelization in data modeling, which allows the model to fit the training data more effectively, leading to higher accuracy on large data sets and higher performance overall. In this research, we developed a novel approach that effectively deployed tools such as mpi and mpi4py from parallel computing to handle data preprocessing and deep learning modeling processes. As a case study, the technique is applied to COVID-19 data from state of Tennessee, USA. Finally, the effectiveness of our approach is demonstrated by comparing it with existing methods without parallel computing concepts like mpi4py. Our results demonstrate promising outcome for the deployment of parallel computing in modeling to minimize high computational cost.

关键词： High computational cost Performance Data preprocessing mpi mpi4py Machine learning Deep learning Data parallelism Model parallelism

来源：评论

学校读者我要写书评

暂无评论

基于h5py和mpi4py的HDF5数据并行处理的探索

引用

计算机产品与流通 2018年第10期7卷 143-144,211页

作者：宋阳周时强李成范上海大学计算机工程与科学学院

随着层次性数据格式的不断发展,HDF5已经成为一种研究数据存储的通用格式之一。然而受限于HDF5数据文件通常包含大量的数据集这一现状,普通串行处理的方式逐渐不能满足数据处理的要求。本文通过介绍HDF5数据和并行计算的基础,结合目前... 详细信息

随着层次性数据格式的不断发展,HDF5已经成为一种研究数据存储的通用格式之一。然而受限于HDF5数据文件通常包含大量的数据集这一现状,普通串行处理的方式逐渐不能满足数据处理的要求。本文通过介绍HDF5数据和并行计算的基础,结合目前流行的数据处理语言python以及广泛应用于并行计算和HDF5处理的两个软件包——mpi4py和h5py,探索了一种新的对HDF5数据并行计算的方式,该方式具有计算简单、效率高的特点。

关键词：并行计算 HDF5 h5py mpi4py

来源：评论

学校读者我要写书评

暂无评论

OMB-py: python Micro-Benchmarks for Evaluating Performance of mpi Libraries on HPC Systems 36

OMB-Py: Python Micro-Benchmarks for Evaluating Performance o...

引用

36th IEEE International Parallel and Distributed Processing Symposium (IEEE IPDPS)

作者： Alnaasan, Nawras Jain, Arpan Shafi, Aamir Subramoni, Hari Panda, Dhabaleswar K. Ohio State Univ Dept Comp Sci & Engn Columbus OH 43210 USA

ISBN: (纸本)9781665497473

python has become a dominant programming language for emerging areas like Machine Learning (ML), Deep Learning (DL), and Data Science (DS). An attractive feature of python is that it provides easy-to-use programming interface while allowing library developers to enhance performance of their applications by harnessing the computing power offered by High Performance Computing (HPC) platforms. Efficient communication is key to scaling applications on parallel systems, which is typically enabled by the Message Passing Interface (mpi) standard and compliant libraries on HPC hardware. mpi4py is a python-based communication library that provides an mpi-like interface for python applications allowing application developers to utilize parallel processing elements including GPUs. However, there is currently no benchmark suite to evaluate communication performance of mpi4py-and python mpi codes in general-on modern HPC systems. In order to bridge this gap, we propose OMB-py-python extensions to the open-source OSU Micro-Benchmark (OMB) suite-aimed to evaluate communication performance of mpi-based parallel applications in python. To the best of our knowledge, OMBpy is the first communication benchmark suite for parallel python applications. OMB-py consists of a variety of point-to-point and collective communication benchmark tests that are implemented for a range of popular python libraries including Numpy, Cupy, Numba, and pyCUDA. Our evaluation reveals that mpi4py introduces a small overhead when compared to native mpi libraries. We plan to publicly release OMB-py to benefit the python HPC community.

关键词： mpi python mpi4py OMB Benchmarks HPC

来源：评论

学校读者我要写书评

暂无评论

Numba-mpi v1.0: Enabling mpi communication within Numba/LLVM JIT-compiled python code

引用

SOFTWAREX 2024年 28卷

作者： Derlatka, Kacper Manna, Maciej Bulenok, Oleksii Zwicker, David Arabas, Sylwester Jagiellonian Univ Fac Math & Comp Sci Krakow Poland Max Planck Inst Dynam & Selforg Gottingen Germany AGH Univ Krakow Fac Phys & Appl Comp Sci Krakow Poland

The numba-mpi package offers access to the Message Passing Interface (mpi) routines from python code that uses the Numba just-in-time (JIT) compiler. As a result, high-performance and multi-threaded python code may utilize mpi communication facilities without leaving the JIT-compiled code blocks, which is not possible with the mpi4py package, a higher-level python interface to mpi. For debugging or code-coverage analysis purposes, numba-mpi retains full functionality of the code even if the JIT compilation is disabled. The numba-mpi API constitutes a thin wrapper around the C API of mpi and is built around Numpy arrays including handling of non-contiguous views over array slices. Project development is hosted at GitHub leveraging the mpi4py/setup-mpi workflow enabling continuous integration tests on Linux (mpiCH, Openmpi & Intel mpi), macOS (mpiCH & Openmpi) and Windows (MS mpi). The paper covers an overview of the package features, architecture and performance. As of v1.0, the following mpi routines are exposed and covered by unit tests: size/rank, [i]send/[i]recv, wait[all|any], test[all|any], allreduce, bcast, barrier, scatter/[all]gather & wtime. The package is implemented in pure python and depends on numpy, numba and mpi4py (the latter used at initialization and as a source of utility routines only). The performance advantage of using numba-mpi compared to mpi4py is depicted with a simple example, with entirety of the code included in listings discussed in the text. Application of numba-mpi for handling domain decomposition in numerical solvers for partial differential equations is presented using two external packages that depend on numba-mpi: py-pde and pyMPDATA-mpi.

关键词： python mpi Numba JIT Multi-threading LLVM mpi4py

来源：评论

学校读者我要写书评

暂无评论

Simplifying non-contiguous data transfer with mpi for python

引用

JOURNAL OF SUPERCOMPUTING 2023年第17期79卷 20019-20040页

作者： Noelp, Klaus Oden, Lena Fernuniv Tech Comp Sci Hagen Germany

python is becoming increasingly popular in scientific computing. The package mpi for python (mpi4py) allows writing efficient parallel programs that scale across multiple nodes. However, it does not support non-contiguous data via slices, which is a well-known feature of Numpy. In this work, we therefore evaluate several methods to support the direct transfer of non-contiguous arrays in mpi4py. This significantly simplifies the code, while the performance basically stays the same. In a PingPong-, Stencil- and Lattice-Boltzmann-Benchmark, we compare the common manual copying, a Numpy-Copy design and a design that is based on mpi derived datatypes. In one case, the mpi derived datatype design could achieve a speedup of 15% in a Stencil-Benchmark on four compute nodes. Our designs are superior to naive manual copies, but for maximum performance manual copies with pre-allocated buffers or mpi persistent communication will be a better choice.

关键词： mpi4py Strided memory Derived datatypes High-performance computing Parallel programming

来源：评论

学校读者我要写书评

暂无评论

HPC as a Service: A naive model 8

HPC as a Service: A naive model

引用

8th International Conference on Information and Communication Technologies (ICICT)

作者： Imran, Hamza Ali Wazir, Saad Ikram, Ahmed Jamal Ikram, Ataul Aziz Ullah, Hanif Ehsan, Maryam FAST Natl Univ Comp & Emerging Sci EPIC Lab Islamabad Pakistan Riphah Int Univ Islamabad Pakistan Univ Gujrat Informat Technol Dept Gujrat Pakistan

ISBN: (纸本)9781728123349

Applications like Big Data, Machine Learning, Deep Learning and even other Engineering and Scientific research requires a lot of computing power;making High-Performance Computing (HPC) an important field. But access to Supercomputers is out of range from the majority. Nowadays Supercomputers are actually clusters of computers usually made-up of commodity hardware. Such clusters are called Beowulf Clusters. The history of which goes back to 1994 when NASA built a Supercomputer by creating a cluster of commodity hardware. In recent times a lot of effort has been done in making HPC Clusters of even single board computers (SBCs). Although the creation of clusters of commodity hardware is possible but is a cumbersome task. Moreover, the maintenance of such systems is also difficult and requires special expertise and time. The concept of cloud is to provide on-demand resources that can be services, platform or even infrastructure and this is done by sharing a big resource pool. Cloud computing has resolved problems like maintenance of hardware and requirement of having expertise in networking etc. An effort is made of bringing concepts from cloud computing to HPC in order to get benefits of cloud. The main target is to create a system which can develop a capability of providing computing power as a service which to further be referred to as Supercomputer as a service. A prototype was made using Raspberry Pi (RPi) 3B and 3B+ Single Board Computers. The reason for using RPi boards was increasing popularity of ARM processors in the field of HPC

关键词： High Performance Computing Single Board Computers Raspberry Pi Portable Cluster Educational Arm-based micro-cluster Cloud Computing HPCaaS Message Passing Interface mpi4py Parallel & Distributed Computing

来源：评论

学校读者我要写书评

暂无评论

ppython Performance Study

pPython Performance Study

引用

IEEE High Performance Extreme Computing Virtual Conference (HPEC)

作者： Byun, Chansup Arcand, William Bestor, David Bergeron, Bill Gadepally, Vijay Houle, Michael Hubbell, Matthew Jananthan, Hayden Jones, Michael Klein, Anna Michaleas, Peter Milechin, Lauren Morales, Guillermo Mullen, Julie Prout, Andrew Reuther, Albert Rosa, Antonio Samsi, Siddharth Yee, Charles Kepner, Jeremy MIT Cambridge MA 02139 USA

ISBN: (纸本)9798350308600

ppython seeks to provide a parallel capability that provides good speed-up without sacrificing the ease of programming in python by implementing partitioned global array semantics (PGAS) on top of a simple file-based messaging library (pythonmpi) in pure python. ppython follows a SPMD (single program multiple data) model of computation. ppython runs on a single-node (e.g., a laptop) running Windows, Linux, or MacOS operating systems or on any combination of heterogeneous systems that support python, including on a cluster through a Slurm scheduler interface so that ppython can be executed in a massively parallel computing environment. It is interesting to see what performance ppython can achieve compared to the traditional socket-based mpi communication because of its unique file-based messaging implementation. In this paper, we present the point-to-point and collective communication performances of ppython and compare them with those obtained by using mpi4py with Openmpi. For large messages, ppython demonstrates comparable performance as compared to mpi4py.

关键词： Performance python mpi ppython mpi4py

来源：评论

学校读者我要写书评

暂无评论

python and HPC for High Energy Physics Data Analyses 7

Python and HPC for High Energy Physics Data Analyses

引用

7th Workshop on python for High-Performance and Scientific Computing (pyHPC)

作者： Sehrish, S. Kowalkowski, J. Paterno, M. Green, C. Fermilab Natl Accelerator Lab POB 500 Batavia IL 60510 USA

ISBN: (纸本)9781450351249

High level abstractions in python that can utilize computing hardware well seem to be an attractive option for writing data reduction and analysis tasks. In this paper, we explore the features available in python which are useful and efficient for end user analysis in High Energy Physics (HEP). A typical vertical slice of an HEP data analysis is somewhat fragmented: the state of the reduction/analysis process must be saved at certain stages to allow for selective reprocessing of only parts of a generally time-consuming workflow. Also, algorithms tend to to be modular because of the heterogeneous nature of most detectors and the need to analyze different parts of the detector separately before combining the information. This fragmentation causes difficulties for interactive data analysis, and as data sets increase in size and complexity (O10 TiB for a "small" neutrino experiment to the O10 PiB currently held by the CMS experiment at the LHC), data analysis methods traditional to the field must evolve to make optimum use of emerging HPC technologies and platforms. Mainstream big data tools, while suggesting a direction in terms of what can be done if an entire data set can be available across a system and analysed with high-level programming abstractions, are not designed with either scientific computing generally, or modern HPC platform features in particular, such as data caching levels, in mind. Our example HPC use case is a search for a new elementary particle which might explain the phenomenon known as "Dark Matter". Using data from the CMS detector, we will use HDF5 as our input data format, and mpi with python to implement our use case.

关键词： HEP analysis mpi python HPC HDF5 numpy pandas mpi4py

来源：评论

学校读者我要写书评

暂无评论

A Parallel Implementation of the Pandas Framework

A Parallel Implementation of the Pandas Framework

引用

作者： Khan, Saba Hafeez University of Houston

学位级别：M.S.

High-performance is a highly desirable trait for applications today. Companies large and small are migrating their serial applications to parallel versions to reduce execution time and increase efficiency. However, preparing serial applications for parallel processing is not a simple process. Pandas, which is a python library containing rich data structures and tools, is used abundantly in Data Science applications. However, Pandas framework is built for single-core processing and is unable to fully utilize multi-core processors or cluster technology. Because of this limitation, Pandas users are forced to look for other frameworks when working with large quantities of data. This thesis introduces a Parallel-Pandas library which makes the process of parallelizing serial Pandas applications easy and transparent. The Parallel-Pandas library provides Pandas users the ability to upgrade existing applications transparently, by using only a library import. This thesis contains details about the design decisions and implementation of the Parallel-Pandas library. The Parallel-Pandas library is evaluated with unit testing, microbenchmarks, and a real-world application with different datasets. Parallel-Pandas has also been compared with pySpark, a framework that provides parallelism by following the MapReduce structure. The results presented in this paper show that the Parallel-Pandas library has promising potential and delivers performance close to manually parallelized and tuned applications.

关键词： High Performance Computing Pandas mpi Data Science Duplicate Detection Big Data python Parallel-Pandas mpi4py

来源：评论

学校读者我要写书评

暂无评论

Akcelerace aplikací na superpočítači pomocí jazyka python

Akcelerace aplikací na superpočítači pomocí jazyka Pyth...

引用

作者： Čelka, Marek Brno University of Technology

V dnešnej dobe sú už všetky počítače, ktoré používame schopné paralelného spracovávania, ktoré nám výrazne šetrí čas pri riešení náročných probl... 详细信息

V dnešnej dobe sú už všetky počítače, ktoré používame schopné paralelného spracovávania, ktoré nám výrazne šetrí čas pri riešení náročných problémov, ako sú napríklad rôzne vedecké výpočty, simulácie alebo predpovede. Témou tejto práce je akcelerácia výpočtovo náročných aplikácií práve pomocou paralelnému spracovania zadaného problému na superpočítači. Pre jednoduchšie pochopenie danej problematiky ľuďmi, ktorých sa priamo týka (napr. vedci, výzkumníci), bol zvolený programovací jazyk python, ktorý je jednoduchý na pochopenie a mocný zároveň. Prvá časť práce sa venuje zoznámeniu s technikami paralelného spracovania pomocou mikrotestov, ktorých výsledky boli diskutované a využité ďalej v práci. Druhá časť práce sa zaoberá problematikou rekonštrukcie obrazu. Výsledky experimentovania s rôzne veľkými obrázkami sú porovnané v rámci sekvenčnej a paralelnej verzie pri rekonštrukcii obrazu a zápise do súboru. Spracované výsledky sú zhodnotené, diskutované a porovnané medzi sebou. Medzi použité metriky patria čas, zrýchlenie, priepustnosť a latencia.

关键词： mpi mpi4py python numpy superpočítač Anselm formát pgm detektor hrán rekonštrukcia obrazu. mpi mpi4py python numpy supercomputer Anselm format pgm edge detector image reconstruction. Text

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：