检索结果-内蒙古大学图书馆

SunwayMR: A distributed parallel computing framework with convenient data-intensive applications programming

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2017年 71卷 43-56页

作者： Wu, Renke Huang, Linpeng Yu, Peng Zhou, Haojie Shanghai Jiao Tong Univ Dept Comp Sci & Engn 800 Dong Chuan Rd Shanghai 200240 Peoples R China Shanghai Jiao Tong Univ Sch Software 800 Dong Chuan Rd Shanghai 200240 Peoples R China Jiangnan Inst Comp Technol State Key Lab Math Engn & Adv Comp 699 Shanshui East Rd Wuxi 214000 Peoples R China

Managing servers integration to realize distributed data computing framework is an important concern. Regardless of the underlying architecture and the actual distributed system's complexity, such framework gives programmers an abstract view of systems to achieve variously data-intensive applications. However, some state-of-the-art frameworks need too much library dependencies and parameters configuration, or lack extensibility in application programming. Moreover, general framework's precise design is a nontrivial work, which is fraught with challenges of task scheduling, message communication and computing efficiency, etc. To address these problems, we present a general, scalable and programmable parallel computing framework called SunwayMR, which only needs GCC/G++ environment. We argue it from the following aspects: (1) Distributed data partitioning, message communication and task organization are given to support transparent application execution on parallel hardware. By searching threads table of each node, the task gets an idle thread (with preferred node IP address) for executing data partition. A novel communication component, SunwayMRHelper, is employed to merge periodical results synchronously. Through identifying whether current node is master or slave, SunWayMR deals with the periodical task's results differently. (2) As for optimizations, a simple fault tolerance is given to resume data-parallel applications, and thread-level stringstream is utilized to boost computing. To ensure ease-of-use, open Application Programming Interface (API) excerpts can be invoked by various of applications with fewer handwritten code than OpenMPl/MPI. We conduct extensively experimental studies to evaluate the performance of SunwayMR over real-world datasets. Results indicate that SunwayMR (runs on 16 computational nodes) outperforms Spark in various applications, and has good scaling with data sizes, nodes and threads. (C) 2017 Elsevier B.V. All rights reserved.

关键词： Parallel processing Computer software software engineering software development environment and technique Distributed programming and environment SunwayMR

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：