检索结果-内蒙古大学图书馆

G-SD: Achieving Fast Reverse Lookup using Scalable Declustering Layout in Large-Scale file systems

IEEE TRANSACTIONS ON CLOUD COMPUTING 2018年第4期6卷 1017-1030页

作者： Wang, Jun Han, Dezhi Zhang, Junyao Yin, Jiangling Univ Cent Florida Dept EECS Orlando FL 32816 USA Shanghai Maritime Univ Coll Informat Engn Shanghai 201306 Peoples R China

With the increasing popularity of cloud computing, current data centers contain petabytes of data in their datacenters. This requires thousands or tens of thousands of storage nodes at a single site. Node failure in these datacenters is normal instead of a rare situation. As a result, data reliability is a great concern. In order to achieve high reliability, data recovery or node reconstruction is a must. Although extensive research works have investigated how to sustain high performance and high reliability in case of node failure at large scale, a reverse lookup problem, namely finding the list of objects for the failed node is not well-addressed. As the first step of failure recovery, this process has a direct impact to the data recovery/node reconstruction. While existing solutions use metadata traversal or data distribution reversing methods for reverse lookup, which are either time consuming or expensive, the deterministic block placement schemes can achieve fast and efficient reverse lookup easily. However, they are designed for centralized, small-scale storage architectures such as RAID etc. Due to their lacking of scalability, they cannot be directly applied in large-scale storage systems. In this paper, we propose Group-Shifted Declustering (G-SD), a deterministic data layout for multi-way replication. G-SD addresses the scalability issue of our previous Shifted Declustering layout and supports fast and efficient reverse lookup. Our mathematical proofs demonstrate that G-SD is a scalable layout that maintains a high level of data availability. We implement a prototype of G-SD and its reverse lookup function on two open source file systems: Ceph and HDFS. Large scale experiments on the Marmot cluster demonstrate that the average speed of G-SD reverse lookup is more than 5x faster than the reverse lookup speed of existing schemes.

关键词： Data layout aistributed file system group-based shifted declustering parallel file system

来源：评论

学校读者我要写书评

暂无评论

DLFT: Data and Layout Aware Fault Tolerance Framework for Big Data Transfer systems

引用

IEEE ACCESS 2021年 9卷 22939-22954页

作者： Kasu, Preethika Hamandawana, Prince Chung, Tae-Sun Ajou Univ Dept Software & Comp Engn Suwon 16499 South Korea Ajou Univ Dept Artificial Intelligence Suwon 16499 South Korea

Various scientific research organizations generate several petabytes of data per year through computational science simulations. These data are often shared by geographically distributed data centers for data analysis. One of the major challenges in distributed environments is failure;hardware, network, and software might fail at any instant. Thus, high-speed and fault tolerant data transfer frameworks are vital for transferring such large data efficiently between the data centers. In this study, we proposed a bloom filter-based data aware probabilistic fault tolerance (DAFT) mechanism that can handle such failures. We also proposed a data and layout aware mechanism for fault tolerance (DLFT) to effectively handle the false positive matches of DAFT. We evaluated the data transfer and recovery time overheads of the proposed fault tolerance mechanisms on the overall data transfer performance. The experimental results demonstrated that the DAFT and DLFT mechanisms exhibit a maximum of 10% and a minimum of 2% recovery time overhead at 80% and 20% fault points respectively. However, we observed minimum to negligible overhead with respect to the overall data transfer rate.

关键词： Data transfer Fault tolerant systems Fault tolerance Servers Layout Data centers Distributed databases Big data geo-distributed data centers fault tolerance bloom filter parallel file system

来源：评论

学校读者我要写书评

暂无评论

Lessons learned from development and operation of the K computer

引用

parallel COMPUTING 2017年 64卷 12-19页

作者： Shoji, Fumiyoshi RIKEN AICS Operat & Comp Technol Div Chuo Ku 7-1-26Minatojima Minami Machi Kobe Hyogo Japan

We report operational experiences of the K computer which is one of the most powerful supercomputers in the world. The K computer achieved excellent results for system availability, job-filling rate and failure rate. On the other hand, approximately 70% of the unscheduled system stop time was caused by file system failures. We analyzed the reasons for the failures and found that a massive and complex system configuration of the file system is one of the crucial factors for the failures. It revealed many potential bugs in the file system software, and such bugs caused many failures which gave severe impacts to the operation. (C) 2017 Elsevier B.V. All rights reserved.

关键词： The K computer Operation improvement Failure analysis parallel file system

来源：评论

学校读者我要写书评

暂无评论

An adaptive cache coherence protocol specification for parallel input/output systems

引用

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 2004年第6期15卷 533-545页

作者： Garcia-Carballeira, F Carretero, J Calderon, A Perez, JM Garcia, JD Univ Carlos III Madrid Comp Architecture Grp Madrid 28911 Spain

Caching has been intensively used in memory and traditional file systems to improve system performance. However, the use of caching in parallel file systems and I/O libraries has been limited to I/O nodes to avoid cache coherence problems. In this paper, we specify an adaptive cache coherence protocol very suitable for parallel file systems and parallel I/O libraries. This model exploits the use of caching, both at processing and I/O nodes, providing performance increase mechanisms as aggressive prefetching and delayed-write techniques. The cache coherence problem is solved by using a dynamic scheme of cache coherence protocols with different sizes and shapes of granularity. The proposed model is very appropriate for parallel I/O interfaces, as MPI-IO. Performance results, obtained on an IBM SP2, are presented to demonstrate the advantages offered by the cache management methods proposed.

关键词： parallel file system caching cache coherence protocol specification

来源：评论

学校读者我要写书评

暂无评论

Design and Evaluation of MPI file Domain Partitioning Methods under Extent-Based file Locking Protocol

引用

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 2011年第2期22卷 260-272页

作者： Liao, Wei-keng Northwestern Univ Dept Elect Engn & Comp Sci Technol Inst L359 Evanston IL 60208 USA

MPI collective I/O has been an effective method for parallel shared-file access and maintaining the canonical orders of structured data in files. Its implementation commonly uses a two-phase I/O strategy that partitions a file into disjoint file domains, assigns each domain to a unique process, redistributes the I/O data based on their locations in the domains, and has each process perform I/O for the assigned domain. The partitioning quality determines the maximal performance achievable by the underlying file system, as the shared-file I/O has long been impeded by the cost of file system's data consistency control, particularly due to the conflicted locks. This paper proposes a few file domain partitioning methods designed to reduce lock conflicts under the extent-based file locking protocol. Experiments from four I/O benchmarks on the IBM GPFS and Lustre parallel file systems show that the partitioning method producing minimum lock conflicts wins the highest performance. The benefit of removing conflicted locks can be so significant that more than thirty times of write bandwidth differences are observed between the best and worst methods.

关键词： parallel I/O MPI-IO parallel file system file locking Lustre GPFS

来源：评论

学校读者我要写书评

暂无评论

file-access characteristics of parallel scientific workloads

引用

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 1996年第10期7卷 1075-1089页

作者： Nieuwejaar, N Kotz, D Purakayastha, A Ellis, CS Best, ML DUKE UNIV DEPT COMP SCILEVINE SCI RES CTRDURHAMNC 27706 MIT MEDIA LABCAMBRIDGEMA 02139

Phenomenal improvements in the computational performance of multiprocessors have not been matched by comparable gains in NO system performance. This imbalance has resulted in I/O becoming a significant bottleneck for many scientific applications. One key to overcoming this bottleneck is improving the performance of multiprocessor file systems. The design of a high-performance multiprocessor file system requires a comprehensive understanding of the expected workload, Unfortunately, until recently, no general workload studies of multiprocessor file systems have been conducted. The goal of the CHARISMA project was to remedy this problem by characterizing the behavior of several production workloads, on different machines, at the level of individual reads and writes. The first set of results from the CHARISMA project describe the workloads observed on an Intel iPSC/860 and a Thinking Machines CM-5. This paper is intended to compare and contrast these two workloads for an understanding of their essential similarities and differences, isolating common trends and platform-dependent variances. Using this comparison, we are able to gain more insight into the general principles that should guide multiprocessor file-system design.

关键词： parallel file system workload characterization multiprocessor parallel I/O scientific computing

来源：评论

学校读者我要写书评

暂无评论

A Cost-Effective Distribution-Aware Data Replication Scheme for parallel I/O systems

引用

IEEE TRANSACTIONS ON COMPUTERS 2018年第10期67卷 1374-1387页

作者： He, Shuibing Sun, Xian-He Wuhan Univ Sch Comp Sci Wuhan 430072 Hubei Peoples R China Illinois Inst Technol Dept Comp Sci Chicago IL 60616 USA

As data volumes of high-performance computing applications continuously increase, low I/O performance becomes a fatal bottleneck of these data-intensive applications. Data replication is a promising approach to improve parallel I/O performance. However, most existing strategies are designed based on the assumption that contiguous requests are being served more efficiently than non-contiguous requests, which is not necessarily true in a parallel I/O system. The reason is that the multiple-server data distribution makes the favorable accesses between contiguous requests and non-contiguous ones indeterminate. In this study, we propose CEDA, a cost-effective distribution-aware data replication scheme to better support parallel I/O systems. As logical file access information is inefficient to make replication decisions in a parallel environment, CEDA considers physical data accesses on servers in both data selection and data placement during a parallel replication process. Specifically, CEDA first proposes a distribution-aware cost model to evaluate the file request time with a given data layout, and then it carries out cost-effective data replication based on replication benefit analysis. We have implemented CEDA as a part of the MPI I/O library in light of high portability on top of the OrangeFS file system. By replaying representative benchmarks and a real application, we collected comprehensive experimental results on both HDD- and SSD-based servers and conclude that CEDA can significantly improve parallel I/O system performance.

关键词： parallel I/O system parallel file system data layout data replication data reorganization

来源：评论

学校读者我要写书评

暂无评论

Reference Implementation of Scalable I/O Low-Level API on Intel Paragon

引用

Journal of Computer Science & Technology 1999年第3期14卷 206-223页

作者：孙凝晖 National Research Center for Intelligent Computing systems Institute of Computing Technology Chinese Academy of Sciences Beijing P.R. China

The Scalable I/O (SIO) Initiative's Low-Level Application Pro- gramming Interface (SIO LLAPI)provides file system implementers with a simplelow-Level interface to support high-level parallel I/O interfaces efficiently and ef fectively. This paper describes a reference implementation and the evaluation of the SIO LLAPI on the Intel Paragon multicomputer. The implementation provides the file system structure and striping algorithm compatible with the parallel file system (PFS) of Intel Paragon, and runs either inside the kernel or as a user level library. The scatter-gather addressing read/write, asynchronous I/O, client caching and prefetching mechanism, file access hint mechanism, collective I/O and highly efficient file copy have been implemented. The preliminary experience shows that the SIO LLAPI provides opportunities of significant performance improvement and is easy to implement. Some high level file system interfaces and applications, such as PFS, ADIO and Hartree-Fock application, are also implemented on top of SIO. The performance of PFS is at least the same as that of Intel's native PFS, and in many cases, such as small sequential file access, huge I/O requests and collective I/O, it is stable and much better. The SIO features help to support high level interfaces easily, quickly and more efficiently, and the cache, prefetching, hints are useful to get better performance based on different access models. The scalability and per formance of SIO are limited by the network latency, network scalable bandwidth,memory copy bandwidth, memory size and pattern of I/O requests. The tradeoff between generality and efficiency should be considered in implementation.

关键词： scalable I/O low-level API parallel file system scalability client cache collective I/O Intel Paragon

来源：评论

学校读者我要写书评

暂无评论

Cost-Aware Region-Level Data Placement in Multi-Tiered parallel I/O systems

引用

IEEE TRANSACTIONS ON parallel AND DISTRIBUTED systemS 2017年第7期28卷 1853-1865页

作者： He, Shuibing Wang, Yang Li, Zheng Sun, Xian-He Xu, Chenzhong Wuhan Univ Comp Sch State Key Lab Software Engn Wuhan 430072 Hubei Peoples R China Chinese Acad Sci Shenzhen Inst Adv Technol Xueyuan Blvd 1068 Shenzhen 518055 Peoples R China Western Illinois Univ Sch Comp Sci Macomb IL 61455 USA IIT Dept Comp Sci Chicago IL 60616 USA

Multi-tiered parallel I/O systems that combine traditional HDDs with emerging SSDs mitigate the cost burden of SSDs while benefiting from their superior I/O performance. While a multi-tiered parallel I/O system is promising for data-intensive applications in high-performance (HPC) domains, placing data on each tier of the system to achieve high I/O performance remains a challenge. In this paper, we propose a cost-aware region-level (CARL) data placement scheme in multi-tiered parallel I/O systems. CARL divides a large file into several small regions, and then places regions on different types of servers based on region access costs. CARL includes a static policy S-CARL and a dynamic policy D-CARL. For applications whose I/O access patterns are completely known, S-CARL calculates the region costs within the entire workload duration, and uses a static data placement scheme to selectively place regions on the proper servers. To adapt to applications whose access patterns are unknown in advance, D-CARL uses a dynamic data placement scheme which migrates data among different servers within each time window. We have implemented CARL under MPI-IO library and OrangeFS parallel file system environment. Our evaluation with representative benchmarks and an application shows that CARL is both feasible and able to improve I/O performance significantly.

关键词： parallel I/O system parallel file system data placement solid state drive

来源：评论

学校读者我要写书评

暂无评论

HFS: A performance-oriented flexible file system based on building-block compositions

引用

ACM TRANSACTIONS ON COMPUTER systemS 1997年第3期15卷 286-321页

作者： Krieger, O Stumm, M UNIV TORONTO DEPT ELECT & COMP ENGNTORONTOON M5S 3G4CANADA

The Hurricane file system (HFS) is designed for (potentially large-scale) shared-memory multiprocessors. Its architecture is based on the principle that, in order to maximize performance for applications with diverse requirements, a file system must support a wide variety of file structures, file system policies, and I/O interfaces. files in HFS are implemented using simple building blocks composed in potentially complex ways. This approach yields great flexibility, allowing an application to customize the structure and policies of a file to exactly meet its requirements. As an extreme example, HFS allows a file's structure to be optimized for concurrent random-access write-only operations by 10 threads, something no other file system can do. Similarly, the prefetching, locking, and file cache management policies can all be chosen to match an application's access pattern. In contrast, most parallel file systems support a single file structure and a small set of policies. We have implemented HFS as part of the Hurricane operating system running on the Hector shared-memory multiprocessor. We demonstrate that the flexibility of HFS comes with little processing or I/O overhead. We also show that for a number of file access patterns, HFS is able to deliver to the applications the full I/O bandwidth of the disks on our system.

关键词： customization data partitioning data replication flexibility parallel computing parallel file system

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：