检索结果-内蒙古大学图书馆

Challenges and Solutions for Tracing Storage systems: A Case Study with Spectrum Scale

ACM TRANSACTIONS ON STORAGE 2018年第2期14卷 18-18页

作者： Vef, Marc-Andre Tarasov, Vasily Hildebrand, Dean Brinkmann, Andre Johannes Gutenberg Univ Mainz Data Ctr ZDV Anselm Franz von Bentzel Weg 12 D-55128 Mainz Germany IBM Res 650 Harry Rd San Jose CA 95120 USA

IBM Spectrum Scale's parallel file system General parallel file system (GPFS) has a 20-year development history with over 100 contributing developers. Its ability to support strict POSIX semantics across more than 10K clients leads to a complex design with intricate interactions between the cluster nodes. Tracing has proven to be a vital tool to understand the behavior and the anomalies of such a complex software product. However, the necessary trace information is often buried in hundreds of gigabytes of by-product trace records. Further, the overhead of tracing can significantly impact running applications and file system performance, limiting the use of tracing in a production system. In this research article, we discuss the evolution of the mature and highly scalable GPFS tracing tool and present the exploratory study of GPFS' new tracing interface, FlexTrace, which allows developers and users to accurately specify what to trace for the problem they are trying to solve. We evaluate our methodology and prototype, demonstrating that the proposed approach has negligible overhead, even under intensive I/O workloads and with low-latency storage devices.

关键词： parallel file system GPFS trace analysis performance

来源：评论

学校读者我要写书评

暂无评论

Performance Evaluation of A Infiniband-based Lustre parallel file system

Performance Evaluation of A Infiniband-based Lustre Parallel...

引用

2nd International Conference on Challenges in Environmental Science and Computer Engineering (CESCE)

作者： Wang, Yuan Lu, Yongquan Qiu, Chu Gao, Pengdong Wang, Jintao Commun Univ China Ctr High Performance Comp Beijing 100024 Peoples R China

As one of the most widely used parallel file systems, Lustre plays an important role in High Performance Computing (HPC). In this paper, an infiniband-based Lustre parallel file system is built firstly. And then a series of experiments are executed to evaluate the performance of luster with infiniaband. Some parameters have great influence on the result, such as size record, client count, stripe count and stripe size. Experimental results indicate that the higher is the speed of network the higher aggregation bandwidth luster could provide. Moreover, Client count and stripe count have more impact on the performance of luster than the record size and stripe size. Appropriate record size and stripe size could improve the performance a little, but not significantly. (C) 2011 Published by Elsevier Ltd. Selection and/or peer-review under responsibility of the Intelligent Information Technology Application Research Association.

关键词： Lustre infiniband parallel file system high performance computing performance evaluation

来源：评论

学校读者我要写书评

暂无评论

Accelerating big data analytics on HPC clusters using two-level storage

引用

parallel COMPUTING 2017年 61卷 18-34页

作者： Xuan, Pengfei Ligon, Walter B. Srimani, Pradip K. Ge, Rong Luo, Feng Clemson Univ Sch Comp Clemson SC 29634 USA Clemson Univ Elect & Comp Engn Clemson SC 29634 USA

Data-intensive applications that are inherently I/O bound have become a major workload on traditional high-performance computing (HPC) clusters. Simply employing data intensive computing storage such as HDFS or using parallel file systems available on HPC clusters to serve such applications incurs performance and scalability issues. In this paper, we present a novel two-level storage system that integrates an upper-level in-memory file system with a lower-level parallel file system. The former renders memory-speed high I/O performance and the latter renders consistent storage with large capacity. We build a two level storage system prototype with Tachyon and OrangeFS, and analyze the resulting I/O throughput for typical MapReduce operations. Theoretical modeling and experiments show that the proposed two-level storage delivers higher aggregate I/O throughput than HDFS and OrangeFS and achieves scalable performance for both read and write. We expect this two-level storage approach to provide insights on system design for big data analytics on HPC clusters. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Two-level storage In-memory file system parallel file system Data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

Lessons learned from development and operation of the K computer

引用

parallel COMPUTING 2017年 64卷 12-19页

作者： Shoji, Fumiyoshi RIKEN AICS Operat & Comp Technol Div Chuo Ku 7-1-26Minatojima Minami Machi Kobe Hyogo Japan

We report operational experiences of the K computer which is one of the most powerful supercomputers in the world. The K computer achieved excellent results for system availability, job-filling rate and failure rate. On the other hand, approximately 70% of the unscheduled system stop time was caused by file system failures. We analyzed the reasons for the failures and found that a massive and complex system configuration of the file system is one of the crucial factors for the failures. It revealed many potential bugs in the file system software, and such bugs caused many failures which gave severe impacts to the operation. (C) 2017 Elsevier B.V. All rights reserved.

关键词： The K computer Operation improvement Failure analysis parallel file system

来源：评论

学校读者我要写书评

暂无评论

Cost-Aware Region-Level Data Placement in Multi-Tiered parallel I/O systems

引用

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 2017年第7期28卷 1853-1865页

作者： He, Shuibing Wang, Yang Li, Zheng Sun, Xian-He Xu, Chenzhong Wuhan Univ Comp Sch State Key Lab Software Engn Wuhan 430072 Hubei Peoples R China Chinese Acad Sci Shenzhen Inst Adv Technol Xueyuan Blvd 1068 Shenzhen 518055 Peoples R China Western Illinois Univ Sch Comp Sci Macomb IL 61455 USA IIT Dept Comp Sci Chicago IL 60616 USA

Multi-tiered parallel I/O systems that combine traditional HDDs with emerging SSDs mitigate the cost burden of SSDs while benefiting from their superior I/O performance. While a multi-tiered parallel I/O system is promising for data-intensive applications in high-performance (HPC) domains, placing data on each tier of the system to achieve high I/O performance remains a challenge. In this paper, we propose a cost-aware region-level (CARL) data placement scheme in multi-tiered parallel I/O systems. CARL divides a large file into several small regions, and then places regions on different types of servers based on region access costs. CARL includes a static policy S-CARL and a dynamic policy D-CARL. For applications whose I/O access patterns are completely known, S-CARL calculates the region costs within the entire workload duration, and uses a static data placement scheme to selectively place regions on the proper servers. To adapt to applications whose access patterns are unknown in advance, D-CARL uses a dynamic data placement scheme which migrates data among different servers within each time window. We have implemented CARL under MPI-IO library and OrangeFS parallel file system environment. Our evaluation with representative benchmarks and an application shows that CARL is both feasible and able to improve I/O performance significantly.

关键词： parallel I/O system parallel file system data placement solid state drive

来源：评论

学校读者我要写书评

暂无评论

Atributed consistent hashing for heterogeneous storage systems 18

Atributed consistent hashing for heterogeneous storage syste...

引用

Proceedings of the 27th International Conference on parallel Architectures and Compilation Techniques

作者： Jiang Zhou Yong Chen Weiping Wang Texas Tech University and Chinese Academy of Sciences Beijing China Texas Tech University Chinese Academy of Sciences Beijing China

ISBN: (纸本)9781450359863

Storage systems are critical building blocks of high-end computing systems and data centers. They demand the flexibility to distribute data effectively and provide high I/O performance. The consistent hashing algorithm is widely used in parallel/distributed file systems due to its decentralized design, scalability, and adaptability to node changes. However, it lacks efficiency in a heterogeneous environment where different storage devices, e.g. hard disk drives and solid state drives, co-exist. In this study, we propose an attributed consistent hashing (attributedCH), to overcome this deficiency. AttributedCH manages heterogeneous nodes on a consistent hashing ring and maintains attributes for each node to characterize distinct node features. It divides the hash ring into sectors and selects nodes from the sector with a comprehensive selection strategy. By considering different attributes, attributedCH achieves adaptive and efficient data placement for heterogeneous storage systems. We have carried out extensive evaluations and the evaluation results confirm that the attributedCH overcomes the deficiency of existing consistent hashing algorithms well and is particularly suitable for heterogeneous storage systems.

关键词： heterogeneous storage system consistent hashing data distribution parallel file system

来源：评论

学校读者我要写书评

暂无评论

Dynamic Management of In-memory Storage for Efficiently Integrating Compute- and Data-intensive Computing on HPC systems 17

Dynamic Management of In-memory Storage for Efficiently Inte...

引用

17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

作者： Xuan, Pengfei Luo, Feng Ge, Rong Srimani, Pradip K. Clemson Univ Sch Comp Clemson SC 29634 USA

ISBN: (纸本)9781509066117

In order to boost the performance of data-intensive computing on HPC systems, in-memory computing frameworks, such as Apache Spark and Flink, use local DRAM for data storage. Optimizing the memory allocation to data storage is critical to delivering performance to traditional HPC compute jobs and throughput to data-intensive applications sharing the HPC resources. Current practices that statically configure in-memory storage may leave inadequate space for compute jobs or miss the opportunity to utilize available space for data-intensive applications. In this paper, we explore techniques to dynamically adjust in-memory storage allocation and provide optimum memory to compute jobs. We have developed a dynamic in-memory storage controller, DynIMS, which monitors memory demands of compute tasks in real time and employs a feedback-based control mechanism to adapt the allocation of in-memory storage. We test DynIMS using HPCC and Spark workloads on a HPC cluster. Experimental results show that DynIMS can achieve up to 5X performance improvement compared to systems with static memory allocations.

关键词： component in-memory storage parallel file system dynamic control HPC

来源：评论

学校读者我要写书评

暂无评论

Self-Assessing Processes in HPC Applications: Alternate approach to Master/Slave Processes

Self-Assessing Processes in HPC Applications: Alternate appr...

引用

IEEE International Conference on Innovative Mechanisms for Industry Applications (ICIMIA

作者： Shukla, Sankalp Rajput, Dheeraj Singh Ahirwar, Maniram Indira Gandhi Engn Coll Sagar MP India

ISBN: (纸本)9781509059607

This paper proposes the use of a new approach, i.e., 'Self Assessing Processes' approach in high performance computing (HPC) applications in which all processes identify the section of data to be computed using them. After the successful completion of these computations, each process employs a Message Passing Interface (MPI) I/O to write the results to a file in parallel. This eliminates idle time of processes, making efficient use of modern parallel file architecture systems. Contemporary high performance applications employ Master/Slave model to make use of modern multi-core architectures. The disadvantage of Master/Slave model is that master processes create communication overheads while distributing and gathering information. Also, master and slave processes remain idle for a considerable amount of time waiting for communication. The master-slave model application involves file I/O, consuming excessive amount of time due to sequential reading and writing by the master process. The performance of the proposed approach is verified by using Intel Trace Analyzer and Collector (ITAC).

关键词： HPC Master/Slave model Multi-Core file I/O parallel file system Message Passing Interface I/O Intel Trace Analyzer and Collector (ITAC)

来源：评论

学校读者我要写书评

暂无评论

Analyzing the parallel I/O Severity of MPI Applications 17

Analyzing the Parallel I/O Severity of MPI Applications

引用

17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)

作者： Mendez, Sandra Rexachs, Dolores Luque, Emilio Leibniz Supercomp Ctr LRZ High Performance Syst Div Garching Germany Univ Autonoma Barcelona Comp Architecture & Operating Syst Dept Barcelona Spain

ISBN: (纸本)9781509066117

Performance evaluation of parallel applications plays an important role in High Performance Computing (HPC). This is also applied to parallel I/O performance evaluation, which requires understanding the I/O pattern of the application and having knowledge about the performance capacity of the HPC I/O system. In this paper, we present a methodology to evaluate the I/O performance of parallel applications based on the I/O severity degree. We define the I/O severity concept taking into account the I/O requirements of a parallel application, the mapping of I/O processes and the configuration of the I/O subsystem. Requirements are expressed in units denominated I/O phases, which are defined using the temporal and spatial pattern of different files of the application. Our approach is applied to the I/O kernels of scientific applications such as S3DIO, FLASH-IO and BT-IO on the SuperMUC supercomputer. Experimental results show that our methodology allows us to identify if a parallel application is limited by the I/O subsystem and identifying possible root causes of the I/O problems.

关键词： HPC systems parallel I/O parallel file system I/O Severity I/O system

来源：评论

学校读者我要写书评

暂无评论

Performance Evaluation of A Infiniband-based Lustre parallel file system

引用

Procedia Environmental Sciences 2011年 11卷 316-321页

作者： Yuan Wang Yongquan Lu Chu Qiu Pengdong Gao Jintao Wang High Performance Computing Center Communication University of China Beijing 100024 China Information Engineering School Communication University of China Beijing 100024 China

关键词： Lustre infiniband parallel file system high performance computing performance evaluation

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：