检索结果-内蒙古大学图书馆

9th IEEE International Conference on Big Data (IEEE BigData)

作者： da Silva, Erico Correia Sato, Liria Matsumoto Midorikawa, Edson Toshimi Univ Sao Paulo Escola Politecn Sao Paulo Brazil

ISBN: (纸本)9781665439022

With the exponential volume growth of the data available for scientific and commercial use, more and more Big Data technologies are gaining focus and importance. Directly related to the efficiency of these techniques is the distributed file system used for data persistence, generally based on low-cost computer clusters. However, the environments used today for Big Data are based on file systems restricted to the WORM pattern (write once, read many) lacking POSIX compatibility. This work uses distributed lock management techniques to create a file system that allows random writing for both HPC and Big Data tools. A local write protocol is implemented to leverage the use of local copies of the data during the write process. Experiments were carried out to evaluate the performance of the proposed write protocol and the scalability of the developed file system. From the experimental results, it is possible to conclude that the achieved performance and scalability improvements were obtained by eliminating limitations imposed by HDFS and leveraging local writes.

关键词： distributed file systems Hadoop Big Data distributed lock management

来源：评论

学校读者我要写书评

暂无评论

Research of Cloud Storage Based on Hadoop distributed file System

Research of Cloud Storage Based on Hadoop Distributed File S...

引用

International Conference on Advances in Materials Science and Information Technologies in Industry (AMSITI)

作者： Han, Yongqi Zhang, Yun Yu, Shui Jilin Agr Univ Dept Informat Technol Changchun 130118 Peoples R China Changchun Univ Sci & Technol Dept Informat Engn Changchun 130600 Peoples R China

ISBN: (纸本)9783038350125

This paper discusses the application of cloud computing technology to store large amount of data in agricultural remote training video and other multimedia data, using four computers to build a Hadoop cloud platform, focus on the Hadoop distributed file System (HDFS) principle and file storage, to achieve massive agricultural multimedia data storage.

关键词： Cloud computing Hadoop distributed file systems HDFS file storage

来源：评论

学校读者我要写书评

暂无评论

The Evolution of the Hadoop distributed file System 32

The Evolution of the Hadoop Distributed File System

引用

32nd IEEE International Conference on Advanced Information Networking and Applications (IEEE AINA)

作者： Maneas, Stathis Schroeder, Bianca Univ Toronto Dept Comp Sci Toronto ON Canada

ISBN: (纸本)9781538653951

Frameworks for large-scale distributed data processing, such as the Hadoop ecosystem, are at the core of the big data revolution we have experienced over the last decade. In this paper, we conduct an extensive study of the Hadoop distributed file System (HDFS)'s code evolution. Our study is based on the reports and patch files (patches) available from the official Apache issue tracker (JIRA) and our goal was to make complete use of the entire history of HDFS at the time and the richness of the available data. The purpose of our study is to assist developers in improving the design of similar systems and implementing more solid systems in general. In contrast to prior work, our study covers all reports that have been submitted over HDFS's lifetime, rather than a sampled subset. Additionally, we include all associated patch files that have been verified by the developers of the system and classify the root causes of issues at a finer granularity than prior work, by manually inspecting all 3302 reports over the first nine years, based on a two-level classification scheme that we developed. This allows us to present a different perspective of HDFS, including a focus on the system's evolution over time, as well as a detailed analysis of characteristics that have not been previously studied in detail. Our results indicate that bug reports constitute the most dominant type, having a continuously increasing rate over time. Moreover, the overall scope and complexity of reports and patch files remain surprisingly stable throughout HDFS' lifetime, despite the significant growth the code base experiences over time. Finally, we created a detailed database that includes all reports and patches, along with the key characteristics we extracted.

关键词： distributed file systems Open-Source Code Evolution System Analysis Hadoop file System (HDFS)

来源：评论

学校读者我要写书评

暂无评论

distributed file Load ReBalancing Methodology For Map Reduce System

Distributed File Load ReBalancing Methodology For Map Reduce...

引用

International Conference on Information Communication and Embedded systems (ICICES)

作者： Saraswathi, U. Anbu, S. Anbazhagan, K. PB Coll Engn ME Comp Sci & Engn Madras Tamil Nadu India PB Coll Engn Dept CSE Madras Tamil Nadu India

ISBN: (纸本)9781479938346

Map reduce systems are widely used to solve large computing work. Employing a map reduce system in distributed file system in which there are master node and worker nodes. This master node splits the jobs in to smaller units and submits those jobs to the worker nodes. The worker nodes process the spitted work and send the results to the master node. The master node collects all the results from the worker nodes and consolidates the results and gives response to the client. The nodes may tend to fail in a networked systems. If a node is failed then the jobs running in the node will be moved to other node. The load will be balanced among the worker nodes. The master node takes care of doing this balancing. But there is problem in which load balancing a node comes under heavy load and not able to compute jobs faster. This project addresses the above problem in which the worker nodes itself will have a distributed load rebalancing algorithm among themselves instead of the master node allocates the same. This kind of approach will tends to work faster than the master node reallocates the unit of work.

关键词： Load Balance Map Reduce distributed file systems

来源：评论

学校读者我要写书评

暂无评论

RDMA-based Cooperative Caching for a distributed file System 21

RDMA-based Cooperative Caching for a Distributed File System

引用

21st IEEE International Conference on Parallel and distributed systems ICPADS

作者： Sasaki, Shin Matsumiya, Ryo Takahashi, Kazushi Oyama, Yoshihiro Tatebe, Osamu Univ Electrocommun Dept Informat Tokyo Japan Univ Tsukuba Dept Comp Sci Tsukuba Ibaraki 305 Japan

ISBN: (纸本)9780769557854

Efficient caching of file data is critical in order to achieve high performance in data-intensive applications. However, only a limited amount of memory is usually available to cache files in client nodes even on high-performance computing platforms. Cooperative caching is an approach that enables client nodes to share memory for file caching and thereby provide a large amount of memory for the file cache in the aggregate. Many studies have confirmed the efficacy of applying cooperative caching to distributed file systems. However, to the best of our knowledge, no study has evaluated an implementation of cooperative caching integrated into a modern distributed file system running on a high-speed network. In this paper, we propose a method that improves the performance of a distributed file system oriented to high-performance computing by integrating cooperative caching into it. In the proposed method, the metadata server of the distributed file system maintains information about the cache in all client nodes, and provides clients with the predicted cache location of any requested file. Further, InfiniBand RDMA is utilized to achieve fast cache transfer between the page caches of client nodes. Implementation of the proposed method in the Gfarm distributed file system and measurement of the performance of three real-world data-intensive applications indicate that the proposed method achieves a maximum speedup of 5.8%.

关键词： distributed file systems file caching cooperative caching InfiniBand RDMA data-intensive applications high-performance computing

来源：评论

学校读者我要写书评

暂无评论

Towards building a conflict-free mobile distributed file system

Towards building a conflict-free mobile distributed file sys...

引用

1st International Workshop on Performance Analysis and Enhancement of Wireless Networks

作者： Boukerche, Azzedine Al-Shaikh, Raed A. EXPEC Comp Ctr Dhahran 31311 Saudi Arabia Univ Ottawa SITE PARADISE Res Lab Ottawa ON K1N 6N5 Canada

The rising demand for mobile computing has created a need for an improved file system that supports mobile clients. Current file systems with support for mobility provide availability through file replicas that are cached at the client side. However, mobile clients may experience different obstacles with regards to the local cache, such as the limited network bandwidth, the intermittent connection, and serious conflicts when synchronizing back to the server. In this paper, we propose a novel mobile distributed file system design, which provides high availability and reliable storage for files and guarantees that file operations are executed regardless of concurrency and failure issues. The design is intended to fit mobile clients (e.g. PDAs and cell phones) that have limited storage space and cannot store all of the data they need, and yet require access to these data at all times. We adopt a server-side caching in order to guarantee sufficient caching space to all mobile clients, and to ensure the availability of files in the case of clients' failures. We present our algorithm, describe its implementation, simulate its high availability functions, and report on its performance evaluation using a cluster of workstations. Our simulation results indicate clearly that our algorithm exhibits a significant degree of automation and conflict-free mobile file system. Copyright (C) 2007 John Wiley & Sons, Ltd.

关键词： distributed file systems distributed computing

来源：评论

学校读者我要写书评

暂无评论

IFLustre: Towards Interference-free and Efficient Storage Allocation in distributed file System 30

IFLustre: Towards Interference-free and Efficient Storage Al...

引用

30th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication systems (MASCOTS)

作者： Kim, Sunggon Sung, Dong Kyu Son, Yongseok Seoul Natl Univ Sci & Technol Seoul South Korea Seoul Natl Univ Seoul South Korea Chung Ang Univ Seoul South Korea

ISBN: (纸本)9781665455800

distributed file systems (DFSs) are widely used in large scale computing environments such as cloud computing and high performance computing systems (HPC) where thousands of applications are executed simultaneously. To support applications that produce and process a large amount of data, DFSs manage limited storage resources in the system and allocate the resources to the applications. To efficiently utilize limited storage resources, it is important to consider the I/O characteristics of applications as well as the effect of resource sharing among multiple applications in the system. In this paper, we first perform empirical performance analysis and investigate the effect of I/O interference caused by storage resource allocation. Based on our empirical performance analysis, we propose IFLustre towards interference-free and efficient storage allocation in Lustre file system. IFLustre first utilizes previous execution logs to determine the necessary storage resources for the application and predict the throughput and runtime of the application. Then, it allocates storage resources via file system configurations based on the prediction results and the DFS allocation status. This allows IFLustre to allocate sufficient resources to the application while mitigating interference from resource sharing. The experimental results show that IFLustre can improve the performance by up to 47% compared with Lustre.

关键词： distributed file systems Storage High Performance Computing

来源：评论

学校读者我要写书评

暂无评论

Towards achieving a highly available distributed file system

Towards achieving a highly available distributed file system

引用

9th International Conference on Advanced Communication Technology (ICACT 2007)

作者： Abdalla, Salim Ahmad, Ismail Tat, Ewe Hong Teh, Gim Aik Kee, Yong Lee MMU Selangor 63100 Darul Eshan Malaysia Intel Technol Sdn Bhd PG 12 Bayan Lepas Free Ind Zone Pulau 11900 Malaysia

ISBN: (纸本)9788955191318

Achieving high availability is a premium goal for many distributed rile systems. file replication is a well-known technique that is used to achieve this goal. It generally offers reduced client latencies and increases files availability. The objective of this paper is to achieve high availability of riles through a novel intelligent replication method. To achieve this goal two algorithms are proposed, namely a primary replica assignment algorithm and an intelligent replica placement algorithms. The assignment algorithm uses an intelligent criteria based scheme to decide which one from a number of replicas to be assigned as the primary copy for a set of replicas. The placement algorithm also uses an intelligent criteria based scheme to decide on replication of riles. The two algorithms have different criteria and the combined effect of both will facilitate for higher availability of files within the distributed rile system.

关键词： distributed file systems replication replica placement

来源：评论

学校读者我要写书评

暂无评论

UnifyFS: A User-level Shared file System for Unified Access to distributed Local Storage 37

UnifyFS: A User-level Shared File System for Unified Access ...

引用

37th IEEE International Parallel and distributed Processing Symposium (IPDPS)

作者： Brim, Michael J. Moodyt, Adam T. Lim, Seung-Hwan Miller, Ross Boehm, Swen Stanaviget, Cameron Mohrort, Kathryn M. Oral, Sarp Oak Ridge Natl Lab Oak Ridge TN USA Lawrence Livermore Natl Lab Lawrence KS USA

ISBN: (纸本)9798350337662

We introduce UnifyFS, a user-level file system that aggregates node-local storage tiers available on high performance computing (HPC) systems and makes them available to HPC applications under a unified namespace. UnifyFS employs transparent I/O interception, so it does not require changes to application code and is compatible with commonly used HPC I/O libraries. The design of UnifyFS supports the predominant HPC I/O workloads and is optimized for bulk-synchronous I/O patterns. Furthermore, UnifyFS provides customizable file system semantics to flexibly adapt its behavior for diverse I/O workloads and storage devices. In this paper, we discuss the unique design goals and architecture of UnifyFS and evaluate its performance on a leadership-class HPC system. In our experimental results, we demonstrate that UnifyFS exhibits excellent scaling performance for write operations and can improve the performance of application checkpoint operations by as much as 3x versus a tuned configuration.

关键词： distributed file systems Parallel I/O Parallel systems Storage hierarchies Storage devices

来源：评论

学校读者我要写书评

暂无评论

Long term distributed file reference tracing: Implementation and experience

引用

SOFTWARE-PRACTICE & EXPERIENCE 1996年第6期26卷 705-736页

作者： Mummert, L Satyanarayanan, M School of Computer Science Carnegie Mellon University 5000 Forbes Avenue Pittsburgh PA 15213–3891 U.S.A.

DFSTrace is a system to collect and analyze long term file reference data in a distributed UNIX workstation environment, The design of DFSTrace is unique in that it pays particular attention to the efficiency, extensibility and the logistics of long-term trace data collection in a distributed environment. The components of DFSTrace are a set of kernel hooks, a kernel buffer mechanism, a data extraction agent, a set of collection servers and post-processing tools. Our experience with DFSTrace has been highly positive, Tracing has been virtually unnoticeable, degrading performance 3-7 per cent, depending on the level of detail of tracing. We have collected file reference traces from approximately 30 workstations continuously for over two years. We have implemented a post-processing library to provide a convenient programmer interface to the traces and have created an on-line database of results from a suite of analysis programs to aid trace selection. Our data has been used for a wide variety of purposes, including file system studies, performance measurement and tuning and debugging, Extensions of DFSTrace have enabled its use in applications such as field reliability testing and determining disk geometry. This paper presents the design, implementation and evaluation of DFSTrace and associated tools and describes how they have been used.

关键词： file reference tracing distributed file systems evaluation measurement

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：