检索结果-内蒙古大学图书馆

Formal Definitions and Performance Comparison of Consistency Models for parallel file systems

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 2024年第6期35卷 937-951页

作者： Wang, Chen Mohror, Kathryn Snir, Marc Lawrence Livermore Natl Lab Livermore CA 94550 USA Univ Illinois Dept Comp Sci Urbana IL 61801 USA

The semantics of HPC storage systems are defined by the consistency models to which they abide. Storage consistency models have been less studied than their counterparts in memory systems, with the exception of the POSIX standard and its strict consistency model. The use of POSIX consistency imposes a performance penalty that becomes more significant as the scale of parallel file systems increases and the access time to storage devices, such as node-local solid storage devices, decreases. While some efforts have been made to adopt relaxed storage consistency models, these models are often defined informally and ambiguously as by-products of a particular implementation. In this work, we establish a connection between memory consistency models and storage consistency models and revisit the key design choices of storage consistency models from a high-level perspective. Further, we propose a formal and unified framework for defining storage consistency models and a layered implementation that can be used to easily evaluate their relative performance for different I/O workloads. Finally, we conduct a comprehensive performance comparison of two relaxed consistency models on a range of commonly seen parallel I/O workloads, such as checkpoint/restart of scientific applications and random reads of deep learning applications. We demonstrate that for certain I/O scenarios, a weaker consistency model can significantly improve the I/O performance. For instance, in small random reads that are typically found in deep learning applications, session consistency achieved a 5x improvement in I/O bandwidth compared to commit consistency, even at small scales.

关键词： Synchronization Load modeling Program processors Computational modeling Deep learning Writing Standards Consistency model parallel i/o parallel file system storage consistency

来源：评论

学校读者我要写书评

暂无评论

HARL: Optimizing parallel file systems with Heterogeneity-Aware Region-Level Data Layout

引用

IEEE TRANSACTIONS ON COMPUTERS 2017年第6期66卷 1048-1060页

作者： He, Shuibing Wang, Yang Sun, Xian-He Xu, Chengzhong Wuhan Univ Comp Sch State Key Lab Software Engn Wuhan 430072 Hubei Peoples R China Chinese Acad Sci Shenzhen Inst Adv Technol Xueyuan Blvd 1068 Shenzhen 518055 Peoples R China IIT Dept Comp Sci Chicago IL 60616 USA

parallel file system (PFS) is commonly used in high-end computing systems. With the emergence of solid state drives (SSDs), hybrid PFS, which consists of both HDD and SSD servers, provides a practical I/O system solution for data-intensive applications. However, most existing data layout schemes are inefficient for hybrid PFS due to their unawareness of server heterogeneities and workload changes in different parts of a file. In this study, we propose a heterogeneity-aware region-level data layout scheme, HARL, to improve the data distribution of a hybrid PFS. HARL first divides a file into fine-grained, varying sized regions according to the workload features of an application, then determines appropriate file stripe sizes on servers for each region based on the performance of heterogeneous servers. Furthermore, to further improve the performance of a hybrid PFS, we propose a dynamic region-level layout scheme, HARL-D, which creates multiple replicas for each region and redirects file requests to the proper replicas with the lowest access costs at the runtime. Experimental results of representative benchmarks and a real application show that HARL can greatly improve I/O system performance, and demonstrate the advantages of HARL-D over HARL.

关键词： parallel I/O system parallel file system solid state drive data layout hybrid parallel file system

来源：评论

学校读者我要写书评

暂无评论

Fault tolerant file models for MPI-IO parallel file systems

引用

14th EuroPVM/MPI Conference

作者： Calderon, A. Garcia-Carballeira, F. Isaila, Florin Keller, Rainer Schulz, Alexander Univ Carlos III Madrid Comp Architecture Grp Dept Comp Sci Madrid Spain Univ Stuttgart High Performance Comp Ctr Stuttgart HLRS D-70550 Stuttgart Germany

ISBN: (纸本)9783540754152

parallelism in file systems is obtained by using several independent server nodes supporting one or more secondary storage devices. This approach increases the performance and scalability of the system, but a fault in one single node can make the whole system fail. In order to avoid this problem, data must be stored using some kind of redundant technique, so that it can be recovered in case of failure. Fault tolerance can be provided in I/O systems by using replication or RAID based schemes. However, most of the current systems apply the same technique of fault tolerant at disk or file system level. This paper(1) describes how fault tolerance support can be used by MPI applications based on PVFS version 2 [1], a well-know parallel file system for clusters. This support can be applied to other parallel file systems with many benefits: fault tolerance at file level, flexible definition of new fault tolerance scheme, and dynamic reconfiguration of the fault tolerance policy.

关键词： parallel file system clusters fault-tolerance data declustering reliability

来源：评论

学校读者我要写书评

暂无评论

A Server-Level Adaptive Data Layout Strategy for parallel file systems

A Server-Level Adaptive Data Layout Strategy for Parallel Fi...

引用

26th IEEE International parallel and Distributed Processing Symposium (IPDPS) / Workshop on High Performance Data Intensive Computing

作者： Song, Huaiming Jin, Hui He, Jun Sun, Xian-He Thakur, Rajeev Dawning Informat Ind Ctr Res & Dev Beijing 100193 Peoples R China IIT Dept Comp Sci Chicago IL 60616 USA Argonne Natl Lab Math & Comp Sci Div 9700 S Cass Ave Argonne IL 60439 USA

ISBN: (纸本)9780769546766

parallel file systems are widely used for providing a high degree of I/O parallelism to mask the gap between I/O and memory speed. However, peak I/O performance is rarely attained due to complex data access patterns of applications. Based on the observation that the I/O performance of small requests is often limited by the request service rate, and the performance of large requests is limited by I/O bandwidth, we take into consideration both factors and propose a server-level adaptive data layout strategy. The proposed strategy adopts different stripe sizes for different file servers according to the data access characteristics on each individual server. We let the file servers that can fully utilize bandwidth hold more data, and the file servers that are limited with request service rate hold less data. As a result, heavy-load servers can offload some data accesses to light-load servers for potential improvement of I/O performance. We present a method to measure access cost for each data block and then utilize an equal-depth histogram approach to distributed data blocks across multiple servers adaptively, so as to balance data accesses on all file servers. Analytical and experimental results demonstrate that the proposed server-level adaptive layout strategy can improve I/O performance by as much as 80.3% and is more appropriate for applications with complex data access patterns.

关键词： Server-level adaptive data layout variable stripe size equal-depth histogram data layout optimization parallel file system

来源：评论

学校读者我要写书评

暂无评论

PSA: A Performance and Space-Aware Data Layout Scheme for Hybrid parallel file systems

PSA: A Performance and Space-Aware Data Layout Scheme for Hy...

引用

DISCS The International Workshop on Data-Intensive Scalable Computing systems

作者： He, Shuibing Liu, Yan Sun, Xian-He Wuhan Univ Sch Comp Wuhan Hubei Peoples R China IIT Dept Comp Sci Chicago IL 60616 USA

ISBN: (纸本)9781479970384

The underlying storage of hybrid parallel file systems (PFS) is composed of both SSD-based file servers (SServer) and HDD-based file servers (HServer). Unlike a traditional HServer, an SServer consistently provides improved storage performance but lacks storage space. However, most current data layout schemes do not consider the differences in performance and space between heterogeneous servers, and may significantly degrade the performance of the hybrid PFSs. In this paper, we propose PSA, a novel data layout scheme, which maximizes the hybrid PFSs performance by applying adaptive varied-size file stripes. PSA dispatches data on heterogeneous file servers not only based on storage performance but also storage space. We have implemented PSA within OrangeFS, a popular parallel file system in the HPC domain. Our extensive experiments using a representative benchmark show that PSA provides superior I/O throughput than the default and performance-aware file data layout schemes.

关键词： parallel I/O system parallel file system Data Layout Solid State Drive

来源：评论

学校读者我要写书评

暂无评论

A Migratory Heterogeneity-Aware Data Layout Scheme for parallel file systems 32

A Migratory Heterogeneity-Aware Data Layout Scheme for Paral...

引用

32nd IEEE International parallel and Distributed Processing Symposium (IPDPS)

作者： He, Shuibing Sun, Xian-He Wang, Yang Xu, Chengzhong Wuhan Univ Sch Comp Sci Wuhan 430072 Hubei Peoples R China IIT Dept Comp Sci Chicago IL 60616 USA Chinese Acad Sci Shenzhen Inst Adv Technol Shenzhen Peoples R China

ISBN: (纸本)9781538643686

parallel file systems (PFSs) are widely deployed to speed up the performance of high-performance computing (HPC) applications. In recent years, hybrid PFSs that consist of HDD-SSD servers, have attracted much attention in HPC community. However, existing data layout schemes do not well consider the characteristics of heterogeneous servers and heterogeneous access patterns, thus may experience considerable inefficiencies. In this study, we propose MHA, a migratory heterogeneity-aware data layout scheme to improve the data distribution of hybrid PFS. More specifically, to accommodate heterogeneous access patterns, MHA first migrates file data into several regions, each with similar access patterns. Then, by leveraging a data access cost model, MHA determines the appropriate stripe sizes on heterogeneous servers to get the best performance on each region. We have implemented MHA under MPI-IO library on top of OrangeFS file system. Experimental results show that MHA can significantly improve the hybrid PFS I/O system performance compared to existing data layout schemes.

关键词： parallel I/O system parallel file system Solid State Drive Data Layout

来源：评论

学校读者我要写书评

暂无评论

The use of locality information on data intensive parallel file systems

The use of locality information on data intensive parallel f...

引用

IEEE 16th International Conference on Computational Science and Engineering (CSE)

作者： Sugawara Junior, Ricardo Ryoiti Sato, Liria Matsumoto Univ Sao Paulo Dept Comp & Digital Syst Engn Sao Paulo Brazil

Many recent data intensive parallel systems builds with cost effective hardware and combine compute and storage facilities. Since bandwidth-bisecting networks are the norm, distributing jobs near data provides significant performance improvements. However, the data locality information is not easily available to the programmer. It requires interaction with file system internals, or the adoption of a custom programming and run-time frameworks that provide locality-aware job scheduling, such as Mapreduce and Hadoop. In this paper, we present a parallel file system implementation combined with virtual files presented as text that can be queried for locality data or written to control the placement of new data blocks. This simplifies how software that do not adhere to Mapreduce's model can benefit from computing near the data. To evaluate the proposed approach, a number of tests were run on an initial implementation using fast disks, with locality-aware cases showing from 2 to 9 times faster reads and higher processor utilization.

关键词： Data Intensive Locality parallel file system Hadoop Virtual file system

来源：评论

学校读者我要写书评

暂无评论

Performance-Aware Data Placement in Hybrid parallel file systems

Performance-Aware Data Placement in Hybrid Parallel File Sys...

引用

14th International Conference on Algorithms and Architectures for parallel Processing (ICA3PP)

作者： He, Shuibing Sun, Xian-He Feng, Bo Feng, Kun Wuhan Univ Sch Comp Wuhan 430072 Hubei Peoples R China Illinois Inst Technol Dept Comp Sci Chicago IL USA

ISBN: (纸本)9783319111971;9783319111964

Hybrid parallel file systems (PFS), which consist of both HDD and SSD servers, provide a promising solution for data-intensive applications. In this study, we propose a performance-aware data placement (PADP) strategy to enable efficient data layout in hybrid PFSs. The basic idea of PADP is to dispatch data on different file servers with adaptive varied-size file stripes based on the server storage performance. By using an effective data access cost model and a linear programming optimization method, the appropriate stripe sizes for each file server are determined effectively. We have implemented PADP within OrangeFS, a widely used parallel file system in HPC domain. Experimental results of representative benchmark show that PADP can significantly improve the I/O performance of hybrid PFSs.

关键词： parallel I/O system parallel file system Solid State Drive

来源：评论

学校读者我要写书评

暂无评论

QoSFM: QoS Support for Metadata I/O In parallel file systems

QoSFM: QoS Support for Metadata I/O In Parallel File Systems

引用

15th IEEE International Conference on High Performance Computing and Communications (HPCC) /11th IEEE/IFIP International Conference on Embedded and Ubiquitous Computing (EUC)

作者： Xiao, Limin Xie, Ke Li, Guoying Ruan, Li Li, Xiuqiao Beihang Univ State Key Lab Software Dev Environm Sch Comp Sci & Engn Beijing 100191 Peoples R China

ISBN: (纸本)9780769550886

Large parallel file systems (PFSs) must service multiple independent workloads concurrently. Workloads often require different quality of service (QoS) of metadata I/O. However, existing PFSs are unable to differentiate metadata I/O from different workloads. There are many researches on QoS support of storage system, they do not focus on metadata I/O of PFSs as one metadata I/O can have several sub-operations and different sub-operations have different access size. In addition, users cannot specify metadata QoS requirements without a suitable QoS specification. This paper presents a novel scheme QoSFM (QoS Support for Metadata) to support metadata I/O QoS in PFSs. QoSFM allows users to specify workloads' metadata QoS requirements through our two-layered QoS specification mechanism. QoSFM can dynamically translate the workload-level goals into system resource needs such as service time per time window and sub-operation latency. Client-side controllers throttle the metadata I/O based on service time allocation to meet the throughput goal while metadata servers (MDSs) reorder the sub-operations based on deadlines to meet the latency goal. Extensive simulations show that QoSFM can effectively satisfy workload-level metadata I/O QoS requirements. Meanwhile, results also show that QoSFM has low overhead.

关键词： QoS Metadata I/O parallel file system Resource allocation

来源：评论

学校读者我要写书评

暂无评论

Early Exploration of Using ChatGPT for Log-based Anomaly Detection on parallel file systems Logs 23

Early Exploration of Using ChatGPT for Log-based Anomaly Det...

引用

32nd International Symposium on High-Performance parallel and Distributed Computing (HPDC) part of the ACM Federated Computing Research Conference (FCRC)

作者： Egersdoerfer, Chris Zhang, Di Dai, Dong Univ N Carolina Charlotte NC 27599 USA

ISBN: (纸本)9798400701559

Log-based anomaly detection has been extensively studied to help detect complex runtime anomalies in production systems. However, existing techniques exhibit several common issues. First, they rely heavily on expert-labeled logs to discern anomalous behavior patterns. But labelling enough log data manually to effectively train deep neural networks may take too long. Second, they rely on numeric model prediction based on numeric vector input which causes model decisions to be largely non-interpretable by humans which further rules out targeted error correction. In recent years, we have witnessed groundbreaking advancements in large language models (LLMs) such as ChatGPT. These models have proven their ability to retain context and formulate insightful responses over entire conversations. They also present the ability to conduct few-shot and in-context learning with reasoning ability. In light of these abilities, it is only natural to explore their applicability in understanding log content and conducting anomaly classification among parallel file system logs.

关键词： anomaly detection parallel file system large language model

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：