检索结果-内蒙古大学图书馆

Application and user-specific data prefetching and parallel read algorithms for distributed file systems

CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2024年第3期27卷 3593-3613页

作者： Nalajala, Anusha Ragunathan, T. Naha, Ranesh Battula, Sudheer Kumar SRM Univ AP Dept CSE Neerukonda 522502 Andhra Pradesh India Sri Ramachandra Inst Higher Educ & Res Fac Engn & Technol Chennai Tamil Nadu India Federat Univ Australia Ctr Smart Analyt Gippsland Campus Churchill Vic 3841 Australia Federat Univ Australia Ctr Smart Analyt Gippsland Campus Churchill Vic 3841 Australia

Cloud computing systems are widely used to deploy big data-based applications because of their high storage and computation capacity. The key component for storage in cloud computing environment is distributed file system which can store and process data produced by big data-based applications effectively. The users of such big data-based applications issue read requests more frequently when compared to write requests. So, most of these cloud-based applications demand optimal performance from the distributed file system, especially for read operations. Numerous caching and prefetching techniques have been proposed in the existing literature to enhance the performance of distributed file system. However, these techniques typically adopt a synchronous approach, focusing on either application data prefetching or user data prefetching, when the user application starts executing and this may result in an extended read access time. Furthermore, the data is prefetched either based on access frequency or reuse distance with out considering the access recency of data which may result in less cache hit ratio. In this paper, we have proposed application-specific and user-specific data prefetching algorithms for prefetching the data from the distributed file system and storing the same in the multi-level caches present in the distributed file system based on the combination of access frequency and recency ranking of file blocks that were previously accessed by client application programs. Additionally, we have divided the cache into two partitions namely user and application caches to store the prefetched data as per the popularity value calculated by considering user and application level accesses. We have also introduced a parallel read algorithm to read data simultaneously from the multiple caches present in the distributed file system environment. The simulation results demonstrate that, the proposed algorithms improved the distributed file systems performance by minimum of

关键词： Big data applications Cloud computing distributed file systems Prefetching techniques Multi-level caches Cache replacement

来源：评论

学校读者我要写书评

暂无评论

Load Rebalancing for distributed file systems in Clouds

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed systems 2013年第5期24卷 951-962页

作者： Hsiao, Hung-Chang Chung, Hsueh-Yi Shen, Haiying Chao, Yu-Chang Natl Cheng Kung Univ Dept Comp Sci & Informat Engn Tainan 70101 Taiwan Natl Cheng Kung Univ Dept Comp Sci & Informat Engn Distributed Comp Res Lab Tainan 70101 Taiwan Clemson Univ Holcombe Dept Elect & Comp Engn Clemson SC 29634 USA Ind Technol Res Inst Cloud Serv Applicat Ctr Cloud Platform Technol Dept Tainan 709 Taiwan

distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions;a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system;that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a large-scale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment.

关键词： Load balance distributed file systems clouds

来源：评论

学校读者我要写书评

暂无评论

State of the Art in distributed file systems: Increasing Performance

State of the Art in Distributed File Systems: Increasing Per...

引用

2nd Eastern European Regional Conference on the Engineering of Computer Based systems (ECBS-EERC)

作者： Bzoch, Pavel Safarik, Jiri Univ W Bohemia Dept Comp Sci & Engn Plzen Czech Republic

ISBN: (纸本)9780769544182

Need of storing huge amounts of data has grown over the past years. The data should be stored for future reuse or for sharing among users. Data files can be stored on a local file system or on a distributed file system. A distributed file system provides many advantages such as reliability, scalability, security, etc. This paper shows new trends in these systems with a focus on increasing performance. These include the organization of data and metadata storage, usage of caching, and design of replication algorithms.

关键词： distributed file systems file replication caching performance

来源：评论

学校读者我要写书评

暂无评论

Cloud distributed file systems: a Benchmark of HDFS, Ceph, GlusterFS, and XtremeFS

Cloud Distributed File Systems: a Benchmark of HDFS, Ceph, G...

引用

IEEE Global Conference on Communications (GLOBECOM) / Workshop on Wireless Networking and Control for UAV

作者： Acquaviva, Luca Bellavista, Paolo Corradi, Antonio Foschini, Luca Gioia, Leo Picone, Pasquale Carlo Maiorano Univ Bologna Dipartimento Informat Sci & Ingn Viale Risorgimento 2 I-40136 Bologna Italy Imola Informat Via Selice 66-A I-40026 Imola BO Italy

ISBN: (纸本)9781538647271

Cloud computing nowadays is the cornerstone for all the business applications, mainly because of its high fault tolerance characteristic. High resilience and availability typical of cloud-native applications are achieved using different technologies. Regarding the file system, the main fault tolerant application examples are distributed file systems, such as HDFS, Ceph, GlusterFS, and XtremeFS. These file systems have different architectures and deployment models than the Traditional distributed file systems (TDFSs), such as NFS. The primary goal of this work is to analyze and compare different Cloud distributed file systems (CDFSs) in terms of characteristics, architecture, reliability, and components. As a key feature, the paper benchmarks them considering as use case an IaaS platform.

关键词： distributed file systems Cloud distributed file systems performance analysis performance benchmarking

来源：评论

学校读者我要写书评

暂无评论

AutoCache: Employing Machine Learning to Automate Caching in distributed file systems 35

AutoCache: Employing Machine Learning to Automate Caching in...

引用

IEEE 35th International Conference on Data Engineering (ICDE)

作者： Herodotou, Herodotos Cyprus Univ Technol Limassol Cyprus

ISBN: (纸本)9781728108902

The use of computational platforms such as Hadoop and Spark is growing rapidly as a successful paradigm for processing large-scale data residing in distributed file systems like HDFS. Increasing memory sizes have recently led to the introduction of caching and in-memory file systems. However, these systems lack any automated caching mechanisms for storing data in memory. This paper presents AutoCache, a caching framework that automates the decisions for when and which files to store in, or remove from, the cache for increasing system performance. The decisions are based on machine learning models that track and predict file access patterns from evolving data processing workloads. Our evaluation using real-world workload traces from a Facebook production cluster compares our approach with several other policies and showcases significant benefits in terms of both workload performance and cluster efficiency.

关键词： automated caching distributed file systems

来源：评论

学校读者我要写书评

暂无评论

Opass: Analysis and Optimization of Parallel Data Access on distributed file systems 29

Opass: Analysis and Optimization of Parallel Data Access on ...

引用

29th IEEE International Parallel and distributed Processing Symposium (IPDPS)

作者： Yin, Jiangling Wang, Jun Zhou, Jian Lukasiewicz, Tyler Huang, Dan Zhang, Junyao Univ Cent Florida Dept Elect Engn & Comp Sci Orlando FL 32816 USA

ISBN: (纸本)9781479986484

In this paper, we study parallel data access on distributed file systems, e.g, the Hadoop file system. Our experiments show that parallel data read requests are often served data remotely and in an imbalanced fashion. This results in a serious disk access and data transfer contention on certain cluster/storage nodes. We conduct a complete analysis on how remote and imbalanced read patterns occur and how they are affected by the size of the cluster. We then propose a novel method to Optimize Parallel Data Access on distributed file systems referred to as Opass. The goal of Opass is to reduce remote parallel data accesses and achieve a higher balance of data read requests between cluster nodes. To achieve this goal, we represent the data read requests that are issued by parallel applications to cluster nodes as a graph data structure where edges weights encode the demands of data locality and load capacity. Then we propose new matching-based algorithms to match processes to data based on the configurations of the graph data structure so as to compute the maximum degree of data locality and balanced access. Our proposed method can benefit parallel data-intensive analysis with various parallel data access strategies. Experiments are conducted on PRObEs Marmot 128-node cluster testbed and the results from both benchmark and well-known parallel applications show the performance benefits and scalability of Opass.

关键词： Parallel Data Access distributed file systems Bipartite Matching

来源：评论

学校读者我要写书评

暂无评论

Survey of distributed file systems: Concepts, Implementations, and Challenges 14

Survey of Distributed File Systems: Concepts, Implementation...

引用

14th International Conference on Information and Communication Technology Convergence, ICTC 2023

作者： Pham, Kiet Tuan Lee, Sangjin Nguyen, Lan Anh Paek, Jeongyeup Son, Yongseok Chung-Ang University Department of Computer Science and Engineering Korea Republic of

ISBN: (纸本)9798350313277

Modern computing now relies heavily on distributed file systems (DFS), which make it possible to store, manage, and retrieve data across remote contexts in an effective manner. Therefore, it is essential to comprehend the DFS environment to optimize data management and storage solutions. We explore DFS in this thorough overview work, discussing its foundational ideas, well-known applications, related difficulties, and potential future research avenues. © 2023 IEEE.

关键词： cloud computing data-intensive computing distributed file systems

来源：评论

学校读者我要写书评

暂无评论

Video file Allocation for Wear-Leveling in distributed Storage systems With Heterogeneous Solid-State-Disks (SSDs)

引用

IEEE TRANSACTIONS ON CIRCUITS AND systems FOR VIDEO TECHNOLOGY 2023年第5期33卷 2477-2490页

作者： Lee, Dayoung Lee, Joonho Song, Minseok Inha Univ Dept Comp Engn Incheon 22212 South Korea SK Hynix Seongnam 13558 South Korea

With the advent of new large-capacity solid-state disks (SSDs) such as quad-level-cells (QLC), SSD arrays can be effectively used in video storage systems that require large-capacity storage space. Typically, SSD manufacturers specify a drive-writes-per-day (DWPD) metric, which is the ratio of bytes written per day to the total capacity in bytes, to ensure an SSD's specified lifetime;it is important to limit the number of write operations by considering the DWPD for each SSD. We propose a new video file allocation technique to effectively manage the heterogeneous DWPD characteristics of SSDs in distributed storage systems. To express the degree of wear-leveling for heterogeneous SSDs, we first introduce the concept of ADWD, which is the actual number of bytes written per day compared to DWPD. We then propose two algorithms for file placement and migration. The file placement algorithm places files greedily based on the bandwidth-to-space ratio (BSR) of each file and SSD to balance the bandwidth usage and storage of the SSD. The file migration algorithm moves files from overloaded to underloaded SSDs to meet bandwidth limit requirements while minimizing the overall ADWD as a result of migration, and then migrates additional popular files to improve SSD bandwidth utilization. To use these algorithms in actual distributed file systems, we implemented a suite of tools for file placement and migration in the Hadoop distributed file system (HDFS). Experimental results show that the proposed algorithm reduces the mean of ADWD by 35.44% and its standard deviation by 69.78% compared to the benchmark methods on average.

关键词： Streaming media Bandwidth Servers Ash Arrays distributed databases Costs Storage management distributed file systems streaming media multimedia systems

来源：评论

学校读者我要写书评

暂无评论

SI-CL-SDEO algorithm for improving HDFS performance and data reliability

引用

RESULTS IN ENGINEERING 2025年 26卷

作者： Dhinakaran, D. Raja, S. Edwin Ramesh, T. Thevahi, B. Prabaharan, G. Vel Tech Rangarajan Dr Sagunthala R&D Inst Sci & T Dept Comp Sci & Engn Chennai India RMK Engn Coll Dept Comp Sci & Engn Chennai India PSR Engn Coll Dept Artificial Intelligence & Data Sci Sivakasi India

In distributed file systems, optimizing data block storage and replication is essential for improving system reliability and system performance. Current optimization strategies often fall short in trading off critical metrics such as data availability, execution time, fault tolerance, and network utilization. To resolve these issues, we suggest that the SI-CL-SDEO (Swarm Intelligence-Chaotic Leader and Salp Differential Evolution Optimization) algorithm be particularly designed for the Hadoop distributed file System (HDFS). The aim is to build an optimization strategy, which can greatly enhance the efficiency and stability of HDFS. The SI-CL-SDEO combines salp swarm optimization and differential evolution algorithms to focus on superior performance across multiple parameters. The performance of the proposed algorithm is evaluated through a wide-ranging set of metrics, such as data availability, execution time, fault tolerance, packet delivery ratio (PDR), network utilization, and average delay for reads and writes. Comparative performance analysis under different load conditions showed that, compared with other existing strategies SI-CL-SDEO obtained high performance. The Key performance highlights include a 40 % higher data availability at high replication factors, a 20 % reduction in execution time under low to medium load conditions, a 5-10 % improvement in fault tolerance over current methods, a 96.98 % optimization of network traffic under low load conditions, and a 160 ms average read latency and a 162 ms average write latency under medium load with higher replication factors. These outcomes confirm that the SI-CL-SDEO method is effective in enhancing HDFS's dependability and performance, giving it a robust solution to the demands of modern distributed data storage.

关键词： distributed file systems HDFS Salp differential evolution optimization Data availability Network utilization

来源：评论

学校读者我要写书评

暂无评论

Analyzing the stability, efficiency, and cost of a dynamic data replica balancing architecture for HDFS

引用

ANNALS OF TELECOMMUNICATIONS 2025年 1-17页

作者： Fazul, Rhauani Weber Aita Mendizabal, Odorico Machado Barcelos, Patricia Pitthan Fed Univ Santa Catarina UFSC Postgrad Program Comp Sci Florianopolis SC Brazil Fed Univ Santa Maria UFSM Dept Languages & Comp Syst Santa Maria Rio Grande do S Brazil

Hadoop distributed file System (HDFS) is known for its specialized strategies and policies tailored to enhance replica placement. This capability is critical for ensuring efficient and reliable access to data replicas, particularly as HDFS operates best when data are evenly distributed within the cluster. In this paper, we build upon earlier practical evaluations and conduct a thorough analysis of the replica balancing process in HDFS, focusing on two critical performance metrics: stability and efficiency. We evaluated these aspects alongside balancing operational cost by contrasting them with conventional HDFS solutions and employing a novel dynamic architecture for data replica balancing. On top of that, we delve into the optimizations in data locality brought about by effective replica balancing and their benefits for data-intensive applications, including enhanced read performance. Our findings reveal the extent to which data imbalance in HDFS directly affects the file system and highlight the struggles of the default replica placement policy in maintaining cluster balance. We examined the real but intricate and temporary effectiveness of on-demand balancing, underscoring the importance of regular and adaptable balancing interventions. This reaffirms the significance of context-aware replica balancing, as provided by the proposed dynamic architecture, not only for maintaining data equilibrium but also for ensuring efficient system performance.

关键词： Replica balancing Data replication Stability Efficiency Data locality distributed file systems

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：