检索结果-内蒙古大学图书馆

1st International Conference on Advances in Electrical, Electronics and Computational Intelligence, ICAEECI 2023

作者： Sabitha, R. Sydulu, Sk. John Karthik, S. Kavitha, M.S. SNS College of Technology Department of Computer Science and Engineering Coimbatore India

ISBN: (纸本)9798350342796

This paper deals the scalable, flexible, and cost-effective solutions. distributed file systems play a crucial role in the design and evolution of cloud storage systems, enabling across multiple servers and data centers. This research paper explores the design principles, key components, and evolution of distributed file systems in the context of cloud storage. We examine the challenges faced by distributed file systems, explore their evolution over time, and discuss emerging trends and future directions in this field. © 2023 IEEE.

关键词： Cloud storage design distributed file systems evolution over time

来源：评论

学校读者我要写书评

暂无评论

Achieving Load Balance for Parallel Data Access on distributed file systems

引用

IEEE TRANSACTIONS ON COMPUTERS 2018年第3期67卷 388-402页

作者： Huang, Dan Han, Dezhi Wang, Jun Yin, Jiangling Chen, Xunchao Zhang, Xuhong Zhou, Jian Ye, Mao Univ Cent Florida Dept Elect Engn & Comp Sci Orlando FL 32816 USA Shanghai Maritime Univ Merchant Marine Coll Shanghai 201306 Peoples R China

The distributed file system, HDFS, is widely deployed as the bedrock for many parallel big data analysis. However, when running multiple parallel applications over the shared file system, the data requests from different processes/executors will unfortunately be served in a surprisingly imbalanced fashion on the distributed storage servers. These imbalanced access patterns among storage nodes are caused because a). unlike conventional parallel file system using striping policies to evenly distribute data among storage nodes, data-intensive file system such as HDFS store each data unit, referred to as chunk file, with several copies based on a relative random policy, which can result in an uneven data distribution among storage nodes;b). based on the data retrieval policy in HDFS, the more data a storage node contains, the higher probability the storage node could be selected to serve the data. Therefore, on the nodes serving multiple chunk files, the data requests from different processes/executors will compete for shared resources such as hard disk head and network bandwidth, resulting in a degraded I/O performance. In this paper, we first conduct a complete analysis on how remote and imbalanced read/write patterns occur and how they are affected by the size of the cluster. We then propose novel methods, referred to as Opass, to optimize parallel data reads, as well as to reduce the imbalance of parallel writes on distributed file systems. Our proposed methods can benefit parallel data-intensive analysis with various parallel data access strategies. Opass adopts new matching-based algorithms to match processes to data so as to compute the maximum degree of data locality and balanced data access. Furthermore, to reduce the imbalance of parallel writes, Opass employs a heatmap for monitoring the I/O statuses of storage nodes and performs HM-LRU policy to select a local optimal storage node for serving write requests. Experiments are conducted on PRObE's Marmot 128-no

关键词： Parallel data access distributed file systems HDFS bipartite matching

来源：评论

学校读者我要写书评

暂无评论

Server-side prefetching in distributed file systems

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2016年第2期28卷 294-310页

作者： Liao, Jianwei Southwest Univ China Coll Comp & Informat Sci Tianshen Rd 2 Chongqing 400715 Peoples R China

This paper presents a proactive data prefetching mechanism on storage servers for distributed file systems to achieve better input/output (I/O) performance. This mechanism requires keeping tracks of what blocks were accessed and the information about client I/O requests to predict which blocks will be accessed in the future. Except for showing a linear access tendency, I/O access operations on blocks also reveal a random or similar-random tendency but with certain access patterns. Thus, to address the prediction for future possible read operations, a linear regression algorithm and a chaotic prediction algorithm have been proposed to forecast the future access requests according to different access patterns. As a result, by employing the information, that is, offsets and request sizes, about the predicted future read requests on disk blocks, the storage servers in our newly presented mechanism can prefetch block data more precisely and then send it to associated client file system proactively. The experimental results illustrate that our proposed mechanism is able to shorten read latency to a great extent and then contribute to better read throughput significantly in certain application contexts. Copyright (c) 2014 John Wiley & Sons, Ltd.

关键词： distributed file systems read operation prediction proactive data prefetching linear regression prediction nonlinear time series

来源：评论

学校读者我要写书评

暂无评论

DMetabench-a metadata benchmark for distributed file systems

引用

JOURNAL OF SUPERCOMPUTING 2011年第2期57卷 179-188页

作者： Biardzki, Christoph Ludwig, Thomas Bayer Akad Wissensch Leibniz Rechenzentrum LRZ D-85748 Garching Germany German Climate Comp Ctr DKRZ D-20146 Hamburg Germany

The performance of metadata processing in large distributed file systems currently presents larger challenges than scaling of data throughput. The paper presents a novel, distributed benchmark called DMetabench for measuring the performance of metadata operations. DMetabench runs in environments with potentially thousands of nodes and allows an assessment of the scalability of metadata operations. Additionally, precise run-time performance data is preserved which allows for a better understanding of performance artifacts. Example results from production file systems are provided and discussed. Possible applications of knowledge about metadata performance scaling include the choice of an optimal parallelization strategy for metadata-intensive workloads in a specific runtime environment.

关键词： distributed file systems Metadata Benchmark

来源：评论

学校读者我要写书评

暂无评论

Adaptive Replica Synchronization for distributed file systems

引用

IEEE systems JOURNAL 2015年第3期9卷 865-877页

作者： Liao, Jianwei Li, Li Chen, Huaidong Liu, Xiaoyan Southwest Univ China Coll Comp & Informat Sci Chongqing 400715 Peoples R China

This paper presents an adaptive replica synchronization mechanism among storage servers (SSs) without the interference from the metadata server (MDS) in a distributed file system. This mechanism employs a chunk list data structure, which holds the information about the relevant chunk replicas and is stored on the associated SSs corresponding to the replicas. Combined with version-based update replay technique, our proposed mechanism can ensure data consistency from the viewpoint of clients and help the SSs to conduct replica synchronization proactively with a per-chunk granularity. In contrast to conventional approaches of replica synchronization that are reliant on the MDS to trigger the synchronization, our proposed mechanism can side step the cumbersome centralized synchronization process to accomplish the same synchronization job. Moreover, a deferred update approach for replica synchronization has been adopted by the newly proposed mechanism for contributing to I/O data rate in write-intensive workloads. The evaluation experiments show that this newly presented mechanism can achieve attractive I/O performance enhancement with less synchronization overhead in specific application contexts.

关键词： Adaptive replica synchronization deferred update distributed file systems I/O data rate version-based update replay

来源：评论

学校读者我要写书评

暂无评论

Performing Initiative Data Prefetching in distributed file systems for Cloud Computing

引用

IEEE TRANSACTIONS ON CLOUD COMPUTING 2017年第3期5卷 550-562页

作者： Liao, Jianwei Trahay, Francois Xiao, Guoqiang Li, Li Ishikawa, Yutaka Southwest Univ China Coll Comp & Informat Sci Chongqing 400715 Peoples R China Telecom SudParis Evry France Southwest Univ China Sch Comp & Informat Sci Chongqing Peoples R China Southwest Univ China Dept Comp Sci Chongqing Peoples R China Univ Tokyo Dept Comp Sci Tokyo Japan

This paper presents an initiative data prefetching scheme on the storage servers in distributed file systems for cloud computing. In this prefetching technique, the client machines are not substantially involved in the process of data prefetching, but the storage servers can directly prefetch the data after analyzing the history of disk I/O access events, and then send the prefetched data to the relevant client machines proactively. To put this technique to work, the information about client nodes is piggybacked onto the real client I/O requests, and then forwarded to the relevant storage server. Next, two prediction algorithms have been proposed to forecast future block access operations for directing what data should be fetched on storage servers in advance. Finally, the prefetched data can be pushed to the relevant client machine from the storage server. Through a series of evaluation experiments with a collection of application benchmarks, we have demonstrated that our presented initiative prefetching technique can benefit distributed file systems for cloud environments to achieve better I/O performance. In particular, configuration- limited client machines in the cloud are not responsible for predicting I/O access operations, which can definitely contribute to preferable system performance on them.

关键词： Mobile cloud computing distributed file systems time series server-side prediction initiative data prefetching

来源：评论

学校读者我要写书评

暂无评论

A General-Purpose Architecture for Replicated Metadata Services in distributed file systems

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed systems 2017年第10期28卷 2747-2759页

作者： Stamatakis, Dimokritos Tsikoudis, Nikos Micheli, Eirini Magoutis, Kostas Brandeis Univ Dept Comp Sci Waltham MA 02435 USA Univ Ioannina Dept Comp Sci & Engn Ioannina 45110 Greece

A large class of modern distributed file systems treat metadata services as an independent system component, separately from data servers. The availability of the metadata service is key to the availability of the overall system. Given the high rates of failures observed in large-scale data centers, distributed file systems usually incorporate high-availability (HA) features. A typical approach in the development of distributed file systems is to design and develop metadata services from the ground up, at significant cost in terms of complexity and time, often leading to functional shortcomings. Our motivation in this paper was to improve on this state of things by defining a general-purpose architecture for HA metadata services (which we call RMS) that can be easily incorporated and reused in new or existing file systems, reducing development time. Taking two prominent distributed file systems as case studies, PVFS and HDFS, we developed RMS variants that improve on functional shortcomings of the original HA solutions, while being easy to build and test. Our extensive evaluation of the RMS variant of HDFS shows that it does not incur an overall performance or availability penalty compared to the original implementation.

关键词： distributed file systems high availability system recovery metadata services

来源：评论

学校读者我要写书评

暂无评论

When Blockchain Meets distributed file systems: An Overview, Challenges, and Open Issues

引用

IEEE ACCESS 2020年 8卷 50574-50586页

作者： Huang, Huawei Lin, Jianru Zheng, Baichuan Zheng, Zibin Bian, Jing Sun Yat Sen Univ Sch Data & Comp Sci Guangzhou 510006 Peoples R China Sun Yat Sen Univ Natl Engn Res Ctr Digital Life Guangzhou 510006 Peoples R China

Constructing globally distributed file systems (DFS) has received great attention. Traditional Peer-to-Peer (P2P) distributed file systems have inevitable drawbacks such as instability, lacking auditing and incentive mechanisms. Thus, Inter-Planetary file System (IPFS) and Swarm, as the representative DFSs which integrate with blockchain technologies, are proposed and becoming a new generation of distributed file systems. Although the blockchain-based DFSs successfully provide adequate incentives and security guarantees by exploiting the advantages of blockchain, a series of challenges, such as scalability and privacy issues, are also constraining the development of the new generation of DFSs. Mainly focusing on IPFS and Swarm, this paper conducts an overview of the rationale, layered structure and cutting-edge studies of the blockchain-based DFSs. Furthermore, we also identify their challenges, open issues and future directions. We anticipate that this survey can shed new light on the subsequent studies related to blockchain-based distributed file systems.

关键词： Peer-to-peer computing distributed databases Web servers file systems Bandwidth Routing Blockchain distributed file systems IPFS swarm

来源：评论

学校读者我要写书评

暂无评论

Block Placement in distributed file systems Based on Block Access Frequency

引用

IEEE ACCESS 2018年 6卷 38411-38420页

作者： Liao, Jianwei Cai, Zhigang Trahay, Francois Peng, Xiaoning Southwest Univ Coll Comp & Informat Sci Chongqing 400715 Peoples R China Huaihua Univ Sch Comp Sci & Engn Huaihua 418000 Peoples R China Telecom SudParis F-91000 Evry France Huaihua Univ Key Lab Intelligent Informat Technol Huaihua 418000 Peoples R China

This paper proposes a new data placement policy to allocate data blocks across storage servers of the distributed/parallel file systems, for yielding even block access workload distribution. To this end, we first analyze the history of block access sequence of a specific application and then introduce a k-partition algorithm to divide data blocks into multiple groups, by referring their access frequency. After that, each group has almost the same access workloads, and we can thus distribute these block groups onto storage servers of the distributed file system, to achieve the goal of uniformly assigning data blocks when running the application. In summary, this newly proposed data placement policy can yield not only an even data distribution but also the block data access balance. The experimental results show that the proposed scheme can greatly reduce I/O time and better improve utilization of storage servers when running the database-relevant applications, compared with the commonly used block data placement strategy, i.e., the round-robin placement policy.

关键词： distributed file systems k-partition block access balance data re-distribution I/O performance

来源：评论

学校读者我要写书评

暂无评论

A Highly Reliable Metadata Service for Large-Scale distributed file systems

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed systems 2020年第2期31卷 374-392页

作者： Zhou, Jiang Chen, Yong Wang, Weiping He, Shuibing Meng, Dan Chinese Acad Sci Inst Informat Engn Beijing 100093 Peoples R China Texas Tech Univ Dept Comp Sci Lubbock TX 79401 USA Zhejiang Univ Coll Comp Sci & Technol Hangzhou 310058 Zhejiang Peoples R China

Many massive data processing applications nowadays often need long, continuous, and uninterrupted data accesses. distributed file systems are used as the back-end storage to provide the global namespace management and reliability guarantee. Due to increasing hardware failures and software issues with the growing system scale, metadata service reliability has become a critical issue as it has a direct impact on file and directory operations. Existing metadata management mechanisms can provide fault tolerance capability to some level but are inadequate. They often have limitations in system availability, state consistence, and performance overhead and lack an effective mechanism to offer metadata reliability. This paper introduces a novel highly reliable metadata service to address these issues in large-scale file systems. Different from traditional strategies, this proposed reliable metadata service adopts a new active-standby architecture for fault tolerance and uses a holistic approach to improve file system availability. A new shared storage pool (SSP) is designed for transparent metadata synchronization and replication between active and standby servers. Based on the SSP, a new policy called multiple actives multiple standbys (MAMS) is presented to perform metadata service recovery in case of failures. A new global state recovery strategy and a smart client fault tolerance mechanism are achieved to maintain the continuity of metadata service. We have implemented such highly reliable metadata service in a prototype file system CFS (Clover file system) and conducted extensive tests to evaluate it. Experimental results confirm that it can significantly improve file system reliability with fast failover under different failure scenarios while having negligible influence on performance. Compared with typical reliability designs in Hadoop Avatar, Hadoop HA, and Boom-FS file systems, the mean-time-to-recovery (MTTR) with the highly reliable metadata service was reduced

关键词： Metadata Servers Protocols Synchronization Fault tolerance Fault tolerant systems distributed file systems metadata service metadata reliability fault tolerance shared metadata storage

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：