DtCraft: A High-Performance Distributed Execution Engine at Scale

作     者：Huang, Tsung-Wei Lin, Chun-Xun Wong, Martin D. F. 

作者机构：Univ Illinois Dept Elect & Comp Engn Champaign IL 61801 USA 

出 版 物：《IEEE TRANSACTIONS ON COMPUTER-AIDED DESIGN OF INTEGRATED CIRCUITS AND SYSTEMS》 (IEEE集成电路与系统的计算机辅助设计汇刊)

年 卷 期：2019年第38卷第6期

页      面：1070-1083页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：National Science Foundation [CCF-1421563  CCF-171883] 

主　　题：Distributed computing parallel programming 

摘      要：Recent years have seen rapid growth in data-driven distributed systems, such as Hadoop MapReduce, Spark, and Dryad. However, the counterparts for high-performance or compute-intensive applications including large-scale optimizations, modeling, and simulations are still nascent. In this paper, we introduce DtCraft, a modern C++ based distributed execution engine to streamline the development of high-performance parallel applications. Users need no understanding of distributed computing and can focus on high-level developments, leaving difficult details, such as concurrency controls, workload distribution, and fault tolerance handled by our system transparently. We have evaluated DtCraft on both micro-benchmarks and large-scale optimization problems, and shown the promising performance from single multicore machines to clusters of computers. In a particular semiconductor design problem, we achieved 30x speedup with 40 nodes and 15x less development efforts over hand-crafted implementation.