HyperQueue: Efficient and ergonomic task graphs on HPC clusters

作     者：Beranek, Jakub Bohm, Ada Palermo, Gianluca Martinovic, Jan Jansik, Branislav 

作者机构：VSB Tech Univ Ostrava IT4innovat Ostrava Czech Republic Politecn Milano DEIB Milan Italy 

出 版 物：《SOFTWAREX》 (SoftwareX)

年 卷 期：2024年第27卷

核心收录：

基　　金：LIGATE project European High-Performance Computing Joint Undertaking (JU) European Union Ministry of Education, Youth and Sports of the Czech Republic [MC2102] H2020 - Industrial Leadership Funding Source: H2020 - Industrial Leadership 

主　　题：Distributed computing Task scheduling High performance computing Job manager 

摘      要：Task graphs are a popular method for defining complex scientific simulations and experiments that run on distributed and HPC (High-performance computing) clusters, because they allow their authors to focus on the problem domain, instead of low-level communication between nodes, and also enable quick prototyping. However, executing task graphs on HPC clusters can be problematic in the presence of allocation managers like PBS or Slurm, which are not designed for executing a large number of potentially short-lived tasks with dependencies. To make task graph execution on HPC clusters more efficient and ergonomic, we have created HYPERQUEUE, an open-source task graph execution runtime tailored for HPC use-cases. It enables the execution of large task graphs on top of an allocation manager by aggregating tasks into a smaller amount of PBS/Slurm allocations and dynamically load balances tasks amongst all available nodes. It can also automatically submit allocations on behalf of the user, it supports arbitrary task resource requirements and heterogeneous HPC clusters, it is trivial to deploy and does not require elevated privileges.