检索结果-内蒙古大学图书馆

21st IEEE International Conference on Parallel and distributed systems ICPADS

作者： Sasaki, Shin Matsumiya, Ryo Takahashi, Kazushi Oyama, Yoshihiro Tatebe, Osamu Univ Electrocommun Dept Informat Tokyo Japan Univ Tsukuba Dept Comp Sci Tsukuba Ibaraki 305 Japan

ISBN: (纸本)9780769557854

Efficient caching of file data is critical in order to achieve high performance in data-intensive applications. However, only a limited amount of memory is usually available to cache files in client nodes even on high-performance computing platforms. Cooperative caching is an approach that enables client nodes to share memory for file caching and thereby provide a large amount of memory for the file cache in the aggregate. Many studies have confirmed the efficacy of applying cooperative caching to distributed file systems. However, to the best of our knowledge, no study has evaluated an implementation of cooperative caching integrated into a modern distributed file system running on a high-speed network. In this paper, we propose a method that improves the performance of a distributed file system oriented to high-performance computing by integrating cooperative caching into it. In the proposed method, the metadata server of the distributed file system maintains information about the cache in all client nodes, and provides clients with the predicted cache location of any requested file. Further, InfiniBand RDMA is utilized to achieve fast cache transfer between the page caches of client nodes. Implementation of the proposed method in the Gfarm distributed file system and measurement of the performance of three real-world data-intensive applications indicate that the proposed method achieves a maximum speedup of 5.8%.

关键词： distributed file systems file caching cooperative caching InfiniBand RDMA data-intensive applications high-performance computing

来源：评论

学校读者我要写书评

暂无评论

IFLustre: Towards Interference-free and Efficient Storage Allocation in distributed file System 30

IFLustre: Towards Interference-free and Efficient Storage Al...

引用

30th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication systems (MASCOTS)

作者： Kim, Sunggon Sung, Dong Kyu Son, Yongseok Seoul Natl Univ Sci & Technol Seoul South Korea Seoul Natl Univ Seoul South Korea Chung Ang Univ Seoul South Korea

ISBN: (纸本)9781665455800

distributed file systems (DFSs) are widely used in large scale computing environments such as cloud computing and high performance computing systems (HPC) where thousands of applications are executed simultaneously. To support applications that produce and process a large amount of data, DFSs manage limited storage resources in the system and allocate the resources to the applications. To efficiently utilize limited storage resources, it is important to consider the I/O characteristics of applications as well as the effect of resource sharing among multiple applications in the system. In this paper, we first perform empirical performance analysis and investigate the effect of I/O interference caused by storage resource allocation. Based on our empirical performance analysis, we propose IFLustre towards interference-free and efficient storage allocation in Lustre file system. IFLustre first utilizes previous execution logs to determine the necessary storage resources for the application and predict the throughput and runtime of the application. Then, it allocates storage resources via file system configurations based on the prediction results and the DFS allocation status. This allows IFLustre to allocate sufficient resources to the application while mitigating interference from resource sharing. The experimental results show that IFLustre can improve the performance by up to 47% compared with Lustre.

关键词： distributed file systems Storage High Performance Computing

来源：评论

学校读者我要写书评

暂无评论

Toward Efficient and Simplified distributed Data Intensive Computing

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed systems 2011年第6期22卷 974-984页

作者： Gu, Yunhong Grossman, Robert Univ Illinois Lab Adv Comp Chicago IL 60607 USA Open Data Grp Chicago IL 60607 USA

While the capability of computing systems has been increasing at Moore's Law, the amount of digital data has been increasing even faster. There is a growing need for systems that can manage and analyze very large data sets, preferably on shared-nothing commodity systems due to their low expense. In this paper, we describe the design and implementation of a distributed file system called Sector and an associated programming framework called Sphere that processes the data managed by Sector in parallel. Sphere is designed so that the processing of data can be done in place over the data whenever possible. Sometimes, this is called data locality. We describe the directives Sphere supports to improve data locality. In our experimental studies, the Sector/Sphere system has consistently performed about 2-4 times faster than Hadoop, the most popular system for processing very large data sets.

关键词： distributed programming distributed file systems performance frameworks

来源：评论

学校读者我要写书评

暂无评论

Adaptive Process Migrations in Coupled Applications for Exchanging Data in Local file Cache

引用

ACM TRANSACTIONS ON AUTONOMOUS AND ADAPTIVE systems 2018年第2期13卷 1–25页

作者： Liao, Jianwei Cai, Zhigang Trahay, Francois Zhou, Jun Xiao, Guoqiang Southwest Univ Coll Comp & Informat Sci Chongqing Peoples R China Telecom SudParis Dept Comp Sci Paris France Nanjing Univ State Key Lab Novel Software Technol Nanjing Jiangsu Peoples R China

Many problems in science and engineering are usually emulated as a set of mutually interacting models, resulting in a coupled or multiphysics application. These component models show challenges originating from their interdisciplinary nature and from their computational and algorithmic complexities. In general, these models are independently developed and maintained, so that they commonly employ the global file system for exchanging their data in the coupled application. To effectively use the local file cache on the compute node for exchanging the data among the processes of such applications, and consequently boosting I/O performance, this article presents a novel mechanism to migrate a process from one compute node to another node on the basis of block I/O dependency. In this newly proposed mechanism, the block I/O dependency between two involved processes running on the different nodes is profiled as block access similarity by taking advantage of the Cohen's kappa statistic. Then, the process is supposed to be dynamically migrated from its source node to the destination node, on which there is another process having heavy block I/O dependency. As a result, both processes can exchange their data by utilizing the local file cache instead of the global file system to reduce I/O time. The experimental results demonstrate that the I/O performance can be significantly improved, and the time required for executing the application can be resultantly decreased, as expected.

关键词： Design Algorithms Performance Coupled application computing process migration distributed file systems block I/O dependency I/O performance

来源：评论

学校读者我要写书评

暂无评论

PRBP: A prioritized replica balancing policy for HDFS balancer

引用

SOFTWARE-PRACTICE & EXPERIENCE 2023年第3期53卷 600-630页

作者： Aita Fazul, Rhauani Weber Barcelos, Patricia Pitthan Fed Univ Santa Maria UFSM Dept Languages & Comp Syst Santa Maria RS Brazil

Data replication is the main fault tolerance mechanism implemented by the Apache Hadoop distributed file System (HDFS). The placement of the data across the cluster directly affects replica balancing and data locality. The HDFS Balancer is the native solution to rebalance the data distribution by moving the blocks from over-utilized to under-utilized nodes. Nevertheless, its current balancing policy does not address the characteristics and specific needs of the applications during data rearrangement. In this work, we present the PRBP, a customized replica balancing policy for the HDFS Balancer. The PRBP is based on a system of priorities, which can be adapted and configured according to different demands of use, either these are related to heterogeneous environments or focused on improving data reliability and availability. The priorities define whether system metrics or aspects of the cluster topology should be considered during the execution of the HDFS Balancer, thus making the process of replica balancing in HDFS more flexible. Based on the priority system, we determine association rules that allow the use of multiple priorities simultaneously. Along with these rules, we present guidelines for using the PRBP as a specialized solution in scenarios that can benefit from reactive replica balancing. In addition, we conducted a practical experimentation to highlight the behavior and the applicability of the guidelines of the PRBP to prioritize replica rearrangement in the file system.

关键词： balancing policy data availability data replication distributed file systems fault tolerance reliability

来源：评论

学校读者我要写书评

暂无评论

Deadline-Aware Cost Optimization for Spark

引用

IEEE TRANSACTIONS ON BIG DATA 2021年第1期7卷 115-127页

作者： Sidhanta, Subhajit Golab, Wojciech Mukhopadhyay, Supratik Indian Inst Technol Jodhpur Jodhpur 342037 Rajasthan India Louisiana State Univ Baton Rouge LA 70803 USA Univ Waterloo Waterloo ON N2L 3G1 Canada

We present OptEx, a closed-form model of job execution on Apache Spark, a popular parallel processing engine. To the best of our knowledge, OptEx is the first work that analytically models job completion time on Spark. The model can be used to estimate the completion time of a given Spark job on a cloud, with respect to the size of the input dataset, the number of iterations, and the number of nodes comprising the underlying cluster. Experimental results demonstrate that OptEx yields a mean relative error of 6 percent in estimating the job completion time. Furthermore, the model can be applied for estimating the cost-optimal cluster composition for running a given Spark job on a cloud under a completion deadline specified in the SLO (i.e., Service Level Objective). We show experimentally that OptEx is able to correctly estimate the required cluster composition for running a given Spark job under a given SLO deadline with an accuracy of 98 percent. We also provide a tool which can classify Spark jobs into job categories based on bisimilarity analysis on lineage graphs collected from the given jobs.

关键词： Sparks Cloud computing Virtual machining Big Data Tools Parallel processing Input variables distributed systems parallel processing distributed file systems middleware performance evaluation reliability availability and serviceability

来源：评论

学校读者我要写书评

暂无评论

Fine Granularity and Adaptive Cache Update Mechanism for Client Caching

引用

IEEE systems JOURNAL 2019年第2期13卷 1596-1607页

作者： Liao, Jianwei Trahay, Francois Cai, Zhigang Xiong, Hailing Chen, Shanxiong Ishikawa, Yutaka Southwest Univ China Coll Comp & Informat Sci Chongqing 400715 Peoples R China Huaihua Univ Coll Comp Sci & Engn Huaihua 418008 Hunan Peoples R China Telecom SudParis F-91000 Evry France RIKEN Wako Saitama 3510198 Japan

distributed file systems have been commonly used as the back end to provide high-performance I/O services for hosting complicated data processing, such as database workloads. By buffering the frequently used data in the local file cache, which is called the client caching mechanism, can reduce the I/O latency and improve the performance of file access. However, the overhead and complexity for ensuring data consistency may offset the pertOrmance benefits caused by data caching. This paper proposes an adaptive, per-block cache update mechanism, which can ensure data consistency of block data in a distributed file system, with low synchronization overhead. Specifically, based on the recent history of READ/WRITE requests, the proposed scheme can adaptively select the best-suited consistency update policy for each data block at run time. The experiments on database applications show that our newly proposed mechanism can greatly reduce the overhead resulted by maintaining cache consistency, especially for the workloads having fluctuating READ/WRITE ratios. As a consequence, the system performance can he also enhanced.

关键词： Adaptive tuning client caching data consistency distributed file systems I/O performance per-block granularity

来源：评论

学校读者我要写书评

暂无评论

distributed Personal Cloud Storage without Third Parties

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed systems 2019年第11期30卷 2434-2448页

作者： Konstantopoulos, Michalis Diamantopoulos, Panos Chondros, Nikos Roussopoulos, Mema Univ Athens Athens 15772 Greece

We present the "It's My Data file system" (IMDfs), a personal, version-tracking, distributed file system that allows users to take custody of their cross-device data management needs, without depending on cloud storage providers. Such a system may be desirable to privacy-conscious users, who worry about the increasing number of cloud storage incidents involving data leakage and corruption, or simply want to synchronize their devices without depending on third parties. IMDfs provides a global, unified view of a user's distributed data collection from any device, user-controlled replication of data across devices, transparent remote access, and support for non-restrictive conflict management. IMDfs achieves all this without sacrificing fault-tolerance or performance and with modest storage overhead. We present the design, implementation, and detailed performance evaluation of IMDfs using benchmarks and traces from real workloads, and we demonstrate that: 1) IMDfs, as a local file system, has reasonable overhead compared with a FUSE loopback file system and Wayback, a local versioning file system, and 2) IMDfs, as a network file system, outperforms NFS and Ori, a state-of-the-art file system for cross-device data management.

关键词： Cloud computing Synchronization Data collection Metadata distributed databases Portable computers Performance evaluation distributed file systems

来源：评论

学校读者我要写书评

暂无评论

Content-dependent chunking for differential compression, the local maximum approach

引用

JOURNAL OF COMPUTER AND SYSTEM SCIENCES 2010年第3-4期76卷 154-203页

作者： Bjorner, Nikolaj Blass, Andreas Gurevich, Yuri Microsoft Res Redmond WA 98052 USA Univ Michigan Dept Math Ann Arbor MI 48109 USA

When a file is to be transmitted from a sender to a recipient and when the latter already has a file somewhat similar to it, remote differential compression seeks to determine the similarities interactively so as to transmit only the part of the new file not already in the recipient's old file Content-dependent chunking means that the sender and recipient chop their files into chunks, with the cutpoints determined by some internal features of the files, so that when segments of the two files agree (possibly in different locations within the files) the cutpoints in such segments tend to be in corresponding locations, and so the chunks agree. By exchanging hash values of the chunks, the sender and recipient can determine which chunks of the new file are absent from the old one and thus need to be transmitted. We propose two new algorithms for content-dependent chunking, and we compare their behavior, on random files, with each other and with previously used algorithms One of our algorithms, the local maximum chunking method, has been implemented and found to work better in practice than previously used algorithms. Theoretical comparisons between the various algorithms can be based on several criteria, most of which seek to formalize the idea that chunks should be neither too small (so that hashing and sending hash values become inefficient) nor too large (so that agreements of entire chunks become unlikely) We propose a new criterion, called the slack of a chunking method, which seeks to measure how much of an interval of agreement between two files is wasted because it lies in chunks that don't agree Finally, we show how to efficiently find the cutpoints for local maximum chunking (C) 2009 Elsevier Inc. All rights reserved

关键词： Networking distributed file systems Compression Ergodic theory

来源：评论

学校读者我要写书评

暂无评论

Forensic implications of stacked file systems

引用

FORENSIC SCIENCE INTERNATIONAL-DIGITAL INVESTIGATION 2024年 48卷

作者： Hilgert, Jan-Niclas Lambertz, Martin Baier, Daniel Fraunhofer FKIE Zanderstr 5 D-53177 Bonn Germany

While file system analysis is a cornerstone of forensic investigations and has been extensively studied, certain file system classes have not yet been thoroughly examined from a forensic perspective. Stacked file systems, which use an underlying file system for data storage instead of a volume, are a prominent example. With the growth of cloud infrastructure and big data, it is increasingly likely that investigators will encounter distributed stacked file systems, such as MooseFS and the Hadoop file System, that employ this architecture. However, current standard models and tools for file system analysis fall short of addressing the complexities of stacked file systems. This paper highlights the forensic challenges and implications associated with stacked file systems, discussing their unique characteristics in the context of forensic analyses. We provide insights through three analyses of different stacked file systems, illustrating their operational details and emphasizing the necessity of understanding this file system category during forensic investigations. For this purpose, we present general considerations that must be made when dealing with the analysis of stacked file systems.

关键词： Storage forensics file systems Stacked file systems distributed file systems moosefs Glusterfs Ecryptfs

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：