检索结果-内蒙古大学图书馆

44th Annual International Conference on parallel Processing Workshops (ICPPW)

作者： He, Shuibing Sun, Xian-He Wang, Yang Kougkas, Antonis Haider, Adnan Wuhan Univ Sch Comp Wuhan 430072 Hubei Peoples R China IIT Dept Comp Sci Chicago IL 60616 USA Natl Univ Def Technol State Key Lab High Performance Comp Changsha 410073 Hunan Peoples R China Chinese Acad Sci Shenzhen Inst Adv Technol Shenzhen Peoples R China

ISBN: (纸本)9781467375887

parallel file systems (PFS) are commonly used in high-end computing systems. With the emergence of solid state drives (SSD), hybrid PFSs, which consist of both HDD and SSD servers, provide a practical I/O system solution for data-intensive applications. However, most existing PFS layout schemes are inefficient for hybrid PFSs due to their lack of awareness of the performance differences between heterogeneous servers and the workload changes between different parts of a file. This lack of recognition can result in severe I/O performance degradation. In this study, we propose a heterogeneity-aware region-level (HARL) data layout scheme to improve the data distribution of a hybrid PFS. HARL first divides a file into fine-grained, varying sized regions according to the changes of an application's I/O workload, then chooses appropriate file stripe sizes on heterogeneous servers based on the server performance for each file region. Experimental results of representative benchmarks show that HARL can greatly improve the I/O system performance.

关键词： parallel I/O system parallel file system Solid State Drive Data Layout

来源：评论

学校读者我要写书评

暂无评论

A probability-based load balancing algorithm for parallel file systems

引用

JOURNAL OF THE CHINESE INSTITUTE OF ENGINEERS 2015年第6期38卷 811-820页

作者： Li, Yong Feng, Dan Shi, Zhan Zheng, Ying China Elect Technol Grp Corp Res Inst 28 Nanjing 210007 Jiangsu Peoples R China Huazhong Univ Sci & Technol Wuhan Natl Lab Optoelect Wuhan 430074 Peoples R China

Load balance is critical for performance in parallel file systems. In those systems, a file is commonly placed in several storage nodes using replication or striping techniques for simultaneous file I/O operations. However, storage nodes may be removed or added as the result of node failure or upgrade. files may also dynamically change with creating, deleting, and appending operations. All of these may result in load imbalance which means some storage nodes will be overloaded through receiving too many requests. Consequently, some nodes will become bottlenecks and degrade overall performance. In this study, we propose a novel load balancing algorithm for parallel file systems, called probability-based load balance (PBLB). The PBLB algorithm aims to achieve the optimal load balance with minimum communication overhead. We implement the PBLB algorithm in the pNFS file system and evaluate it with IOzone benchmark. Experimental results show that the PBLB algorithm is effective and stable, and it outperforms the random algorithm which is a classical load balancing algorithm by 16.2% in load balance and 16.3% in performance. Besides, the experimental results also show that the PBLB algorithm is able to deliver high scalability.

关键词： load balance probability parallel file system

来源：评论

学校读者我要写书评

暂无评论

Seismic data IO and sorting optimization in HPC through ANNs prediction based auto-tuning for ExSeisDat

引用

NEURAL COMPUTING & APPLICATIONS 2023年第8期35卷 5855-5888页

作者： Tipu, Abdul Jabbar Saeed Conbhui, Padraig O. Howley, Enda Univ Galway Sch Comp Sci Galway Ireland Irish Ctr High End Comp Dublin Ireland

ExSeisDat is designed using standard message passing interface (MPI) library for seismic data processing on high-performance super-computing clusters. These clusters are generally designed for efficient execution of complex tasks including large size IO. The IO performance degradation issues arise when multiple processes try accessing data from parallel networked storage. These complications are caused by restrictive protocols running by a parallel file system (PFS) controlling the disks and due to less advancement in storage hardware itself as well. This requires and leads to the tuning of specific configuration parameters to optimize the IO performance, commonly not considered by users focused on writing parallel application. Despite its consideration, the changes in configuration parameters are required from case to case. It adds up to further degradation in IO performance for a large SEG-Y format seismic data file scaling to petabytes. The SEG-Y IO and file sorting operations are the two of the main features of ExSeisDat. This research paper proposes technique to optimize these SEG-Y operations based on artificial neural networks (ANNs). The optimization involves auto-tuning of the related configuration parameters, using IO bandwidth prediction by the trained ANN models through machine learning (ML) process. Furthermore, we discuss the impact on prediction accuracy and statistical analysis of auto-tuning bandwidth results, by the variation in hidden layers nodes configuration of the ANNs. The results have shown the overall improvement in bandwidth performance up to 108.8% and 237.4% in the combined SEG-Y IO and file sorting operations test cases, respectively. Therefore, this paper has demonstrated the significant gain in SEG-Y seismic data bandwidth performance by auto-tuning the parameters settings on runtime by using an ML approach.

关键词： HPC MPI-IO parallel file system Machine learning ANN SEG-Y ExSeisDat

来源：评论

学校读者我要写书评

暂无评论

Accelerating big data analytics on HPC clusters using two-level storage

引用

parallel COMPUTING 2017年 61卷 18-34页

作者： Xuan, Pengfei Ligon, Walter B. Srimani, Pradip K. Ge, Rong Luo, Feng Clemson Univ Sch Comp Clemson SC 29634 USA Clemson Univ Elect & Comp Engn Clemson SC 29634 USA

Data-intensive applications that are inherently I/O bound have become a major workload on traditional high-performance computing (HPC) clusters. Simply employing data intensive computing storage such as HDFS or using parallel file systems available on HPC clusters to serve such applications incurs performance and scalability issues. In this paper, we present a novel two-level storage system that integrates an upper-level in-memory file system with a lower-level parallel file system. The former renders memory-speed high I/O performance and the latter renders consistent storage with large capacity. We build a two level storage system prototype with Tachyon and OrangeFS, and analyze the resulting I/O throughput for typical MapReduce operations. Theoretical modeling and experiments show that the proposed two-level storage delivers higher aggregate I/O throughput than HDFS and OrangeFS and achieves scalable performance for both read and write. We expect this two-level storage approach to provide insights on system design for big data analytics on HPC clusters. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Two-level storage In-memory file system parallel file system Data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

A Holistic Heterogeneity-Aware Data Placement Scheme for Hybrid parallel I/O systems

引用

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 2020年第4期31卷 830-842页

作者： He, Shuibing Li, Zheng Zhou, Jiang Yin, Yanlong Xu, Xiaohua Chen, Yong Sun, Xian-He Zhejiang Univ Coll Comp Sci & Technol Hangzhou 310027 Peoples R China Stockton Univ Sch Business Comp Sci Program Galloway NJ 08205 USA Chinese Acad Sci Inst Informat Engn Beijing 100864 Peoples R China Inst Artificial Intelligence Intelligent Comp Syst Res Ctr Zhejiang Lab Hangzhou 311100 Peoples R China Kennesaw State Univ Dept Comp Sci Kennesaw GA 30144 USA Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA Illinois Inst Technol Dept Comp Sci Chicago IL 60616 USA

We present H2DP, a holistic heterogeneity-aware data placement scheme for hybrid parallel I/O systems, which consist of HDD servers and SSD servers. Most of the existing approaches focus on server performance or application I/O pattern heterogeneity in data placement. H2DP considers three axes of heterogeneity: server performance, server space, and application I/O pattern. More specifically, H2DP determines the optimized stripe sizes on servers based on server performance, keeps only critical data on all hybrid servers and the rest data on HDD servers, and dynamically migrates data among different types of servers at run-time. This holistic heterogeneity-awareness enables H2DP to achieve high performance by alleviating server load imbalance, efficiently utilizing SSD space, and accommodating application pattern variation. We have implemented a prototype of H2DP under MPICH2 atop OrangeFS. Extensive experimental results demonstrate that H2DP significantly improve I/O system performance compared to existing data placement schemes.

关键词： Servers system performance Bandwidth Computer science Distributed databases Sun file systems parallel I O system parallel file system hybrid parallel file system data placement solid state drive

来源：评论

学校读者我要写书评

暂无评论

Design and Implementation of Burst Buffer Over-Subscription Scheme for HPC Storage systems

引用

IEEE ACCESS 2023年 11卷 3386-3401页

作者： Bang, Jiwoo Sim, Alexander Lockwood, Glenn K. K. Eom, Hyeonsang Sung, Hanul Seoul Natl Univ Dept Comp Sci & Engn Seoul 08826 South Korea Lawrence Berkeley Natl Lab Computat Res Div Berkeley CA 94720 USA Sangmyung Univ Dept Game Design & Dev Seoul 03016 South Korea

Burst Buffer is widely used in supercomputer centers to bridge the performance gap between computational power and the high-performance I/O systems. The primary role of Burst Buffer is to temporarily absorb the bursty I/O and reduce the heavy access on parallel file system (PFS). However, the job resource manager on High-Performance Computer (HPC) systems prefers to use a dedicated Burst Buffer allocation approach, which eventually leads to the severely underutilized Burst Buffer resource. To improve the efficiency of using the expensive Burst Buffer resource, we analyze the I/O patterns on Burst Buffer in depth. We propose Burst Buffer over-subscription allocation method, which improves Burst Buffer utilization by allowing each job to access Burst Buffer only during its I/O phases so that the jobs can overlap each other. Furthermore, we develop a new I/O congestion-aware scheduler and a transparent data management system between Burst Buffer and PFS. Our approach also reduces the memory overhead and improves the data persistence of the data management system by adapting the persistent memory. With the proposed approach, not only the Burst Buffer utilization can be improved, but also HPC applications can achieve high I/O performance by exploiting the powerful Burst Buffer hardware capabilities. Experimental results show that BBOS can improve Burst Buffer utilization by up to 120% while more stable and higher checkpoint performance is guaranteed even under high I/O loads compared to other state-of-the-art schedulers. Besides, our approach can improve the hit ratio of restart requests by up to 96.4% and provides up to 210% higher restart throughput on Burst Buffer.

关键词： Resource management Supercomputers Memory management Hardware file systems Engines Writing Storage automation Publishing Burst buffer checkpoint demotion over-subscription parallel file system restart

来源：评论

学校读者我要写书评

暂无评论

Array files for computational chemistry: MP2 energies

引用

JOURNAL OF COMPUTATIONAL CHEMISTRY 2007年第7期28卷 1215-1220页

作者： Ford, Alan R. Janowski, Tomasz Pulay, Peter Univ Arkansas Fulbright Coll Arts & Sci Dept Chem & Biochem Fayetteville AR 72701 USA

A simple message-passing implementation for distributed disk storage, called array files (AF), is described. It is designed primarily for parallelizing computational chemistry applications but it should be useful for any application that handles large amounts of data stored on disk. AF allows transparent distributed storage and access of large data files. An AF consists of a set of logically related records, i.e., blocks of data. It is assumed that the records have the typical dimension of matrices in quantum chemical calculations, i.e., they range from 0.1 to similar to 32 MB in size. The individual records are not striped over nodes;each record is stored on a single node. As a simple application, second-order Moller-Plesset (MP2) energies have been implemented using AF. The AF implementation approaches the efficiency of the hand-coded program. MP2 is relatively simple to parallelize but for more complex applications, such as Coupled Cluster energies, the AF system greatly simplifies the programming effort. (C) 2007 Wiley Periodicals, Inc.

关键词： parallel file system MP2 electron correlation parallel calculation

来源：评论

学校读者我要写书评

暂无评论

ALDM: Adaptive Loading Data Migration in Distributed file systems

引用

IEEE TRANSACTIONS ON MAGNETICS 2013年第6期49卷 2645-2652页

作者： Tan, Zhipeng Zhou, Wei Feng, Dan Zhang, Wenhua Huazhong Univ Sci & Technol Sch Comp Sci Wuhan Natl Lab Optoelect Wuhan 430074 Peoples R China

This paper analyzes the problems of load balance caused by data access in parallel file systems and gives an accurate way to estimate the load of Data Servers. On the base of this evaluation, the paper proposes an adaptive loading data migration (ALDM) strategy to balance the load of data servers. Experiment results show ALDM has little impact on system performance during data migration, and the performance improves about 10% after data migration completed. Furthermore, ALDM also has good adaptability to various loads in different storage systems.

关键词： Data migration load balance load detection parallel file system

来源：评论

学校读者我要写书评

暂无评论

Disk-directed I/O for MIMD multiprocessors

引用

ACM TRANSACTIONS ON COMPUTER systemS 1997年第1期15卷 41-74页

作者： Kotz, D Dartmouth College Hanover NH

Many scientific applications that run on today's multiprocessors, such as weather forecasting and seismic analysis, are bottlenecked by their file-I/O needs. Even if the multiprocessor is configured with sufficient I/O hardware, the file system software often fails to provide the available bandwidth to the application. Although libraries and enhanced file system interfaces can make a significant improvement, we believe that fundamental changes are needed in the file server software. We propose a new technique, dish-directed I/O, to allow the disk servers to determine the flow of data for maximum performance. Our simulations show that tremendous performance gains are possible both for simple reads and writes and for an out-of-core application. Indeed, our disk-directed I/O technique provided consistent high performance that was largely independent of data distribution and obtained up to 93% of peak disk bandwidth. It was as much as 18 times faster than either a typical parallel file system or a two-phase-I/O library.

关键词： collective I/O disk-directed I/O file caching MIMD parallel file system parallel I/O

来源：评论

学校读者我要写书评

暂无评论

An agent architecture for managing data resources in a grid environment

引用

FUTURE GENERATION COMPUTER systemS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2009年第7期25卷 747-755页

作者： Perez, Maria S. Sanchez, Alberto Abawajy, Jemal H. Robles, Victor Pena, Jose M. DATSI FI Univ Politecn Madrid Fac Informat Madrid 28660 Spain Deakin Univ Geelong Vic 3217 Australia

The agent paradigm has been successfully used in a large number of research areas. MAPFS. a parallel file system, constitutes One Successful application of agents to the I/O field, providing a multiagent I/O architecture. The use of a multiagent system implies coordination and cooperation among its agents. MAPFS is oriented to clusters of workstations, where agents are applied in order to provide features Such as caching or prefetching. The adaptation of MAPFS to a grid environment is named MAPFS-Grid. Agents can help to increase the performance of data-intensive applications running on top of the grid. This paper describes the conceptual agent framework and the communication model used in MAPFS-Grid, which provides the management of data resources in a grid environment. The evaluation of our proposal shows the advantages of using agents in a data grid. (C) 2008 Elsevier B.V. All rights reserved.

关键词： Multiagent system Cooperation parallel file system Data grid

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：