检索结果-内蒙古大学图书馆

31st International Symposium on High-Performance parallel and Distributed Computing (HPDC)

作者： Paul, Arnab K. Choi, Jong Youl Karimi, Ahmad Maroof Wang, Feiyi Oak Ridge Natl Lab Oak Ridge TN 37830 USA BITS Pilani K K Birla Goa Campus Zuarinagar Goa India

ISBN: (纸本)9781450391993

Monitoring and analyzing a wide range of I/O activities in an HPC cluster is important in maintaining mission-critical performance in a large-scale, multi-user, parallel storage system. Center-wide I/O traces can provide high-level information and fine-grained activities per application or per user running in the system. Studying such large-scale traces can provide helpful insights into the system. It can be used to develop predictive methods for making predictive decisions, adjusting scheduling policies, or providing decisions for the design of next-generation systems. However, sharing real-world I/O traces to expedite such research efforts leaves a few concerns;i) the cost of sharing the large traces is expensive due to this large size, and ii) privacy concern is an issue. We address such issues by building an end-to-end machine learning (ML) workflow that can generate I/O traces for large-scale HPC applications. We leverage ML based feature selection and generative models for I/O trace generation. The generative models are trained on I/O traces collected by the darshan I/O characterization tool over a period of one year. We present a two-step generation process consisting of two deep-learning models, called the feature generator and the trace generator. The combination of two-step generative models provides robustness by reducing the bias of the model and accounting for the stochastic nature of the I/O traces across different runs of an application. We evaluate the performance of the generative models and show that the two-step model can generate time-series I/O traces with less than 20% root mean square error.

关键词： Darshan Generative Modeling Feature Selection Clustering parallel file system

来源：评论

学校读者我要写书评

暂无评论

Software-Based Disk Power Management for Scientific Applications

Software-Based Disk Power Management for Scientific Applicat...

引用

作者： Son, Seung Woo PennState University Libraries

学位级别：Doctor of Philosophy

Power consumption by high-performance systems is becoming an increasing concern for system designers and software writers a like. Disk subsystem is known to be a major contributor to the overall power budget of high-performance systems. Most scientific applications today rely heavily on disk I/O for out-of-core computations, checkpointing, and data visualization. To reduce excess energy consumption on disk system, prior studies proposed several hardware or OS-based disk power management schemes. While such schemes have been known to be effective in certain cases, they might miss opportunities for better energy savings due to their reactive nature. While compiler based schemes can make more accurate decisions on a given application by extracting disk access patterns statically, the lack of runtime information on the status of shared disks may lead to wrong decisions when multiple applications exercise the same set of disks concurrently. Therefore, quantitative comparison of reactive, compiler-based, and hybrid schemes is very *** dissertation makes four major contributions towards more effective disk power management for scientific applications that use disk-resident data frequently (i.e., so called I/O-intensive applications). First, it shows that, while conventional hardware based disk power management scheme is useful in certain cases, compiler-driven approach can be more effective for array-based scientific applications executing on parallel architectures. Second, it shows that restructuring the application code increases length of disk idle periods, thereby leading to better exploitation of available power-saving capabilities. Third, it proposes a compiler-directed energy-aware prefetching scheme for scientific applications that process disk-resident data sets. Finally, it proposes a runtime system support for software-based disk power management scheme. The proposed runtime system is implemented within PVFS2, a parallel file system. We conclude by a

关键词： Low power compiler disk subsystem parallel file system runtime

来源：评论

学校读者我要写书评

暂无评论

Challenges and Solutions for Tracing Storage systems: A Case Study with Spectrum Scale

引用

ACM TRANSACTIONS ON STORAGE 2018年第2期14卷 18-18页

作者： Vef, Marc-Andre Tarasov, Vasily Hildebrand, Dean Brinkmann, Andre Johannes Gutenberg Univ Mainz Data Ctr ZDV Anselm Franz von Bentzel Weg 12 D-55128 Mainz Germany IBM Res 650 Harry Rd San Jose CA 95120 USA

IBM Spectrum Scale's parallel file system General parallel file system (GPFS) has a 20-year development history with over 100 contributing developers. Its ability to support strict POSIX semantics across more than 10K clients leads to a complex design with intricate interactions between the cluster nodes. Tracing has proven to be a vital tool to understand the behavior and the anomalies of such a complex software product. However, the necessary trace information is often buried in hundreds of gigabytes of by-product trace records. Further, the overhead of tracing can significantly impact running applications and file system performance, limiting the use of tracing in a production system. In this research article, we discuss the evolution of the mature and highly scalable GPFS tracing tool and present the exploratory study of GPFS' new tracing interface, FlexTrace, which allows developers and users to accurately specify what to trace for the problem they are trying to solve. We evaluate our methodology and prototype, demonstrating that the proposed approach has negligible overhead, even under intensive I/O workloads and with low-latency storage devices.

关键词： parallel file system GPFS trace analysis performance

来源：评论

学校读者我要写书评

暂无评论

I/O performance analysis of machine learning workloads on leadership scale supercomputer

引用

PERFORMANCE EVALUATION 2022年 157卷

作者： Karimi, Ahmad Maroof Paul, Arnab K. Wang, Feiyi Oak Ridge Natl Lab Oak Ridge TN 37830 USA BITS Pilani Birla Goa Campus KK Pilani India

The popularity of machine learning technologies and frameworks has led to an in-creasingly large number of machine learning workloads running on high-performance computing (HPC) clusters. The ML workflows are readily being adopted in diverse computational fields such as Biology, Physics, Materials, and Computer Science. The I/O behavior of the emerging ML workloads distinctly differs from the traditional HPC workloads, such as simulation or checkpoint/restart-based HPC I/O behavior. Addition-ally, the ML workloads have also pushed for the utilization of GPUs or a combination of CPUs and GPUs in addition to using only CPUs for computational tasks. The diverse and complex I/O behavior of ML workloads requires extensive study and is critical for the efficient performance of various layers of the I/O stack and the overall performance of HPC workloads. This work aims to fill the gap in understanding the I/O behavior of emerging ML workloads by providing an in-depth analysis of ML jobs running on large-scale leadership HPC systems. In particular, we have analyzed the behavior of jobs based on the scale of the jobs, the science domains, and the processing units used by the ML jobs. The analysis was performed on 23,000 ML jobs collected from one year of Darshan logs running on Summit, which is one of the fastest supercomputers. We also collect the CPU and GPU usage of 15,165 ML jobs by merging the Darshan dataset with the power usage of the processing units on Summit. Therefore, this paper is able to provide a systematic I/O characterization of ML workloads on a leadership scale HPC machine to understand how the I/O behavior differs for workloads across various science domains, the scale of workloads, and processing units and analyze the usage of parallel file system and burst buffer by ML I/O workloads. We have made several observations regarding I/O performances and access patterns through various analytical studies and discuss the important lessons learnt from the perspe

关键词： Burst buffer Darshan I O characterization Machine learning parallel file system Processing units

来源：评论

学校读者我要写书评

暂无评论

Uncovering I/O demands on HPC platforms: Peeking under the hood of Santos Dumont

引用

JOURNAL OF parallel AND DISTRIBUTED COMPUTING 2023年第1期182卷

作者： Carneiro, Andre Ramos Bez, Jean Luca Osthoff, Carla Schnorr, Lucas Mello Navaux, Philippe O. A. Fed Univ Rio Grande Do Sul UFRGS Inst Informat Porto Alegre Brazil Lawrence Berkeley Natl Lab LBNL Berkeley CA 94720 USA Natl Lab Sci Computat LNCC Rio De Janeiro Brazil

High-Performance Computing (HPC) platforms are required to solve the most diverse large-scale scientific problems in various research areas, such as biology, chemistry, physics, and health sciences. Researchers use a multitude of scientific softwares, which have different requirements. These include input and output operations, which directly impact performance due to the existing difference in processing and data access speeds. Thus, supercomputers must efficiently handle mixed workload when storing data from the applications. Understanding the set of applications and their performance running in a supercomputer is paramount to understanding the storage system's usage, pinpointing possible bottlenecks, and guiding optimization techniques. This research proposes a methodology and visualization tool to evaluate a supercomputer's data storage infrastructure's performance, taking into account the diverse workload and demands of the system over a long period of operation. As a study case, we focus on the Santos Dumont supercomputer, identifying inefficient usage, problematic performance factors, and providing guidelines on how to tackle those issues. & COPY;2023 The Author(s). Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons .org /licenses /by-nc -nd /4 .0/).

关键词： Lustre parallel file system High-performance storage I/O characterization Metadata

来源：评论

学校读者我要写书评

暂无评论

A Cached Middleware to Improve I/O Performance Using POSIX Interface

引用

Procedia Computer Science 2016年 85卷 125-132页

作者： S. Ambareesh Jesna Fathima Associate Professor Department of Computer Science and Engineering Vemana Institute of Technology Bengaluru

The performance of computing devices is increasing day by day, the storage devices are not capable to reach with this benchmark. The compute and storage resources are forming two parts of the shared network. This paper is presenting a storage middleware called HyCache+ to reduce the high bi-section bandwidth of the parallel computing systems. The middleware uses Programmable Operating system (POSIX) interface to the end users to swap the data with the high capacity network attached storage. The system uses a 2-layer scheduling approach. The HyCache+ can be deployed on the IBM Blue Gene / P supercomputers for the performance comparison.

关键词： Middleware parallel file system POSIX

来源：评论

学校读者我要写书评

暂无评论

Improving the availability of supercomputer job input data using temporal replication

引用

COMPUTER SCIENCE-RESEARCH AND DEVELOPMENT 2009年第3-4期23卷 149-157页

作者： Wang, Chao Zhang, Zhe Ma, Xiaosong Vazhkudai, Sudharshan S. Mueller, Frank North Carolina State Univ Dept Comp Sci 890 Oval Dr EBII Raleigh NC 27695 USA ORNL Comp Sci & Math Div Oak Ridge TN 37831 USA

Storage systems in supercomputers are a major reason for service interruptions. RAID solutions alone cannot provide sufficient protection as 1) growing average disk recovery times make RAID groups increasingly vulnerable to disk failures during reconstruction, and 2) RAID does not help with higher-level faults such failed I/O nodes. This paper presents a complementary approach based on the observation that files in the supercomputer scratch space are typically accessed by batch jobs whose execution can be anticipated. Therefore, we propose to transparently, selectively, and temporarily replicate "active" job input data by coordinating the parallel file system with the batch job scheduler. We have implemented the temporal replication scheme in the popular Lustre parallel file system and evaluated it with real-cluster experiments. Our results show that the scheme allows for fast online data reconstruction, with a reasonably low overall space and I/O bandwidth overhead.

关键词： Temporal replication Batch job scheduler Reliability Supercomputer parallel file system

来源：评论

学校读者我要写书评

暂无评论

Atributed consistent hashing for heterogeneous storage systems 18

Atributed consistent hashing for heterogeneous storage syste...

引用

Proceedings of the 27th International Conference on parallel Architectures and Compilation Techniques

作者： Jiang Zhou Yong Chen Weiping Wang Texas Tech University and Chinese Academy of Sciences Beijing China Texas Tech University Chinese Academy of Sciences Beijing China

ISBN: (纸本)9781450359863

Storage systems are critical building blocks of high-end computing systems and data centers. They demand the flexibility to distribute data effectively and provide high I/O performance. The consistent hashing algorithm is widely used in parallel/distributed file systems due to its decentralized design, scalability, and adaptability to node changes. However, it lacks efficiency in a heterogeneous environment where different storage devices, e.g. hard disk drives and solid state drives, co-exist. In this study, we propose an attributed consistent hashing (attributedCH), to overcome this deficiency. AttributedCH manages heterogeneous nodes on a consistent hashing ring and maintains attributes for each node to characterize distinct node features. It divides the hash ring into sectors and selects nodes from the sector with a comprehensive selection strategy. By considering different attributes, attributedCH achieves adaptive and efficient data placement for heterogeneous storage systems. We have carried out extensive evaluations and the evaluation results confirm that the attributedCH overcomes the deficiency of existing consistent hashing algorithms well and is particularly suitable for heterogeneous storage systems.

关键词： heterogeneous storage system consistent hashing data distribution parallel file system

来源：评论

学校读者我要写书评

暂无评论

Monitoring Sensor Measurement Anomalies of Streaming Environmental Data Using a Local Correlation Score

Monitoring Sensor Measurement Anomalies of Streaming Environ...

引用

International Conference on Computing for Geospatial Research and Application

作者： Ian Taylor Julia L. Sharp David L. White Jason O. Hallstrom Gene Eidson J. Barr von Oehsen Edward B. Duffy Charles V. Privette Charles T. Cook Aravindh Sampath Gyanas Radhakrishnan Clemson University

ISBN: (纸本)9781479903016

Real-time quality control (QC) of streaming natural resource data is needed to support the delivery of high quality data to system users. QC processes need to enable the identification of aberrations, as well as trends that may indicate degradation or component failures. These QC processes form a framework to support the goal of verified data delivered in a timely manner. In this paper, we investigate a method of computing Local Correlation Score (LCS) to detect anomalous patterns among sensor platforms in a concurrent manner. We use the R programming language and OpenMPI. Using empirical tests, we determine the benefits of computing the LCS in parallel, and on various sizes of clusters. We also analyze its use for real-time mapping of Intelligent River data. Our results show that the LCS computed concurrently is an effective means for prompt quality assurance of natural resource data.

关键词： Real-time data Local Correlation Score Quality control R OpenMPI parallel file system Clusters

来源：评论

学校读者我要写书评

暂无评论

A parallel out-of-core computing system using PVFS for Linux clusters 04

A parallel out-of-core computing system using PVFS for Linux...

引用

Proceedings of the international workshop on Storage network architecture and parallel I/Os

作者： Jianqi Tang Binxing Fang Mingzeng Hu Hongli Zhang Harbin Institute of Technology Harbin China

ISBN: (纸本)9781450378222

Cluster systems become a new and popular approach to parallel computing. More and more scientists and engineers use clusters to solve problems with large sets of data for its high processing power, low price and good scalability. Since the traditional out-of-core programs are difficult to write and the virtual memory system does not perform well, we develop a parallel out-of-core computing system using PVFS named POCCS. POCCS provides convenient interface to write out-of-core codes and the global view of the out-of-core data. The software architecture, data storage model and system implementation are described in this paper. The experimental results show that POCCS extends the problem sizes that can be solved and the performance of POCCS is better than the virtual memory system while the data set is large.

关键词： out-of-core computing Linux cluster out-of-core array parallel file system

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：