检索结果-内蒙古大学图书馆

Optimisation to the execution performance of grid job based on distributed file system

INTERNATIONAL JOURNAL OF PARALLEL EMERGENT AND distributed systemS 2012年第2期27卷 109-121页

作者： Hu, Liang D. Lin, Lin Zhao, Jia Che, Xilong Wei, Xiaohui Jilin Univ Coll Comp Sci & Technol Changchun 130012 Peoples R China

In grid computing, many applications are invoked by the means of job submission. Grid job submission comprises the following scenarios: file stage-in, execution and file stage-out. Most of the grid midwareprovides the job submission infrastructure to help to submit jobs, and GridFTP (filedownloading) is used in the file stage operations. These scenarios exit mostly in batch mode. It causes computational resources waiting for the data transfer and limits the performance of the grid application, especially when the application is data/communication intensive. Using the file sharing of distributed file system instead of GridFTP in the job submission, the scenarios of grid job submission can be processed in parallel, whereas the waiting of computational resource can be eliminated. By this way, the execution performance of the grid job is promoted. After the comparison of typical distributed file systems, network attached storage that meets the requirements of grid is picked out, and the file sharing mode for the data exchange in grid job is proposed. In the tests of a series of data-communicationintensive jobs, the performance promotion is verified. Comparing with GridFTP, the average promotion with Common Internet file system is 16.8% to single-task jobs and 19.5% to multiple-task jobs.

关键词： grid computing grid job CIFS data exchange distributed file system performance optimisation

来源：评论

学校读者我要写书评

暂无评论

BlinkFS: A distributed file system for intermittent power

引用

SUSTAINABLE COMPUTING-INFORMATICS & systemS 2015年 6卷 69-80页

作者： Sharma, Navin Irwin, David Shenoy, Prashant Univ Massachusetts CS Dept Amherst MA 01003 USA Univ Massachusetts ECE Dept Amherst MA 01003 USA

The ability to use intermittent power in data centers introduces numerous new opportunities for optimization, including (i) using real-time markets to buy more power when it is cheap, (ii) increasing the fraction of clean, but intermittent, renewable power, (iii) capping power for long periods to extend UPS lifetime during blackouts, and (iv) fully utilizing a data center's power delivery infrastructure. The capability to run off intermittent power also moves us closer to the vision of a net-zero data center that consumes no net energy from the electric grid and has a small carbon footprint. However, designing systems to operate under intermittent power is challenging, since applications often access persistent distributed state, where power fluctuations can impact data availability and I/O performance. To address the problem, we design and implement BlinkFS, which combines blinking with a power-balanced data layout and popularity-based replication/reclamation to optimize I/O throughput and latency as power varies. Our experiments show that BlinkFS outperforms approaches that co-opt existing energy-proportional distributed file systems (DFSs) for intermittent power, particularly at low steady power levels and high levels of intermittency. For example, we show that BlinkFS reduces completion time for MapReduce-style jobs by 42% at 50% full power compared to existing energy-proportional DFSs. (C) 2014 Elsevier Inc. All rights reserved.

关键词： distributed file system Intermittent power

来源：评论

学校读者我要写书评

暂无评论

Algorithms for Maintaining Consistency of Cached Data for Mobile Clients in distributed file system

引用

INTERNATIONAL JOURNAL OF distributed systemS AND TECHNOLOGIES 2017年第1期8卷 17-33页

作者： Bzoch, Pavel Safarik, Jiri Univ West Bohemia Plzen Czech Republic Univ West Bohemia Fac Sci Appl Dept Comp Sci & Engn Plzen Czech Republic

A cache stores data in order to serve future requests to those data faster. In mobile devices, the data have to be transferred from a server through the mobile network before being stored in the cache. The mobile network is prone to failure caused by users' movements and by the placement of base transceiver stations. Moreover, the mobile devices use various telecommunications technologies and therefore the speed of the network is highly variable. Using a cellular network for communication is also expensive. The cache is an intermediate component which addresses this problem. Once the data are downloaded, they can be stored in the cache for possible future reuse. When using a cache, the system designer presumes that the data will be requested again in the future. On the other hand, the original data stored on the server can be changed. Then, the cached data are in an inconsistent state. In this paper, authors present an adaptive method for maintaining the consistency of cached data which saves network traffic by reducing the number of messages needed for inconsistency detection.

关键词： Cache Client-Side Caching Consistency distributed file system Mobile Devices

来源：评论

学校读者我要写书评

暂无评论

A Model of Erasure-Coding-Based distributed file system

A Model of Erasure-Coding-Based Distributed File System

引用

IEEE Asia Pacific Cloud Computing Congress

作者： Han, Yuanbo Li, Hui Hou, Hanxu Peking Univ Shenzhen Grad Sch Shenzhen Eng Lab Converged Networks Technol Adv Network Technol Lab Shenzhen Peoples R China

ISBN: (纸本)9781467329040

Today's exponential growth in network bandwidth and storage capacity has inspired different classes of distributed storage infrastructures. However replication has become the actual way for storing data across data centers, no matter the system is centralized framework or peer-to-peer infrastructure. Erasure code, which is well researched and more space efficient than replication, has multiple tradeoffs that need to be seriously considered. In this paper, we present an erasure-coding-based model coupled with hadoop library, and we call it the Erasure-Coding-Based distributed file system. We discuss the detailed design of our model and provide some recommendations on other coding strategy for distributed file systems.

关键词： Erasure Code distributed file system Hadoop

来源：评论

学校读者我要写书评

暂无评论

A KIND OF distributed file system BASED ON MASSIVE SMALL fileS STORAGE 2012

A KIND OF DISTRIBUTED FILE SYSTEM BASED ON MASSIVE SMALL FIL...

引用

International Conference on Wavelet Active Media Technology and Information Processing (ICWAMTIP)

作者： Liu, Di Kuang, Shi-Jie Univ Elect Sci & Technol China Fac Comp Sci & Engn Chengdu 611731 Peoples R China

ISBN: (纸本)9781467316842;9781467346856

With the development of network storage systems, distributed file system has become more and more important. In the storage of the large and medium-sized websites, requirement of storage with mass small file is of extremely importance, especially the small pictures, electronic documents and some other kind of small files. Considering the extremely large number of small files, if we choose the traditional distributed file system without additional improvement, the efficiency will be greatly reduced. QFS is such a distributed file system that focuses on storage of massive small files. The file system has made great improvements in many aspects such as the metadata management, high concurrency access, storage efficiency. Moreover, its design is simple and efficient.

关键词： distributed file system Massive Small files Metadata Storage Efficiency

来源：评论

学校读者我要写书评

暂无评论

The Hadoop distributed file system

The Hadoop Distributed File System

引用

26th Symposium on Mass Storage systems and Technologies (MSST)

作者： Shvachko, Konstantin Kuang, Hairong Radia, Sanjay Chansler, Robert Yahoo Sunnyvale CA USA

ISBN: (纸本)9781424471539

The Hadoop distributed file system (HDFS) is designed to store very large data sets reliably, and to stream those data sets at high bandwidth to user applications. In a large cluster, thousands of servers both host directly attached storage and execute user application tasks. By distributing storage and computation across many servers, the resource can grow with demand while remaining economical at every size. We describe the architecture of HDFS and report on experience using HDFS to manage 25 petabytes of enterprise data at Yahoo!.

关键词： Hadoop HDFS distributed file system

来源：评论

学校读者我要写书评

暂无评论

Secure Cloud distributed file system 11

Secure Cloud Distributed File System

引用

11th International Conference on Internet Technology and Secured Transactions (ICITST)

作者： Mar, Kheng Kok Hu, ZhengQing Law, Chee Yong Wang, Meifen Nanyang Polytech Singapore Sch Informat Technol Singapore 569830 Singapore

ISBN: (纸本)9781908320735

Traditional methods of securing data are challenged by specific nature and architecture of cloud. With increasing sophistication of cyber attackers and advancement of cryptanalysis techniques, encryption alone is not sufficient to ensure data security. A more adaptive and flexible approach to data security is thus required. In this paper, we introduce a secure distributed file system which can be layered transparently on existing public cloud storage infrastructure. Our approach combines secret sharing and information dispersal algorithm with more flexible authentication and access control security measures to provide secure storage for enterprise data in the cloud. Our proposed architecture supports a clear separation of roles of storage owner and information owner.

关键词： Cloud Storage distributed file system Secured Storage Information Dispersal Algorithm Secret Sharing Peer-to-Peer

来源：评论

学校读者我要写书评

暂无评论

Improvement of The Integrity Verification Application using Timestamp Mechanism for distributed file system 38

Improvement of The Integrity Verification Application using ...

引用

38th Annual IEEE International Computer Software and Applications Conference (COMPSAC)

作者： Watanabe, Hidenobu W Iwarria, Tsukasa Murata, Ken. T. Ushiyama, Shigenori Muto, Yasushi Natl Inst & Commun Technol NICT Tokyo Japan Japan NUS CO Ltd Tokyo Japan

ISBN: (纸本)9781479935789

A storage system constructed by a wide-area distributed file system of open source software has been requested to ensure the certain existence of the non-falsification data by a trusted third party. We developed an integrity verification application using timestamp mechanism for Gfarm which is a wide-area distributed file system of open source software made in Japan. However, the application remained low performance. This paper explains an improvement method of the application and reports its performance in a basic evaluation.

关键词： distributed file system Scientific Data Archive Integrity Verification Timestamp Mechanism

来源：评论

学校读者我要写书评

暂无评论

A distributed file system for Frequency Reading of Various file Sizes

A Distributed File System for Frequency Reading of Various F...

引用

10th Web Information system and Application Conference (WISA)

作者： Ma, Pengfei Yin, Yanshen Lan, Chao Zhang, Yong Xing, Chunxiao Tsinghua Univ Dept Comp Sci & Technol Beijing 100084 Peoples R China Tsinghua Univ Res Inst Informat Technol Beijing 100084 Peoples R China

ISBN: (纸本)9780769551340

The Web file Management system (WFMS) is a distributed file system, which is designed for frequency reading of various file sizes. It consists of one single master and multi data servers. We use blocks to store data, which contains different size of chunks. The block/chunk mechanism achieves nearly 100% utilization of block. We also design hot block to raise the reading speed of frequency blocks through increasing the replicas automatically. WFMS works on the web exposing RESTful services and providing authority control. Besides, we implement our search engine with Lucene which provides query service by file name, file size, file type, date, owner and so on. Until now, our system has stored over 40TB data which contains more than 2 million files and it has worked stably for two years.

关键词： WFMS distributed file system file system

来源：评论

学校读者我要写书评

暂无评论

SciDFS: An In-situ Processing system for Scientific Array Data based on distributed file system

SciDFS: An In-situ Processing System for Scientific Array Da...

引用

IEEE International Conference on Big Data and Smart Computing (BigComp)

作者： Han, Donghyoung Nam, Yoon-Min Kim, Min-Soo Park, Kyongseok Han, Sunggeun DGIST Informat & Comp Engn Daegu South Korea KISTI Sci Data Res Ctr Daejeon South Korea

ISBN: (纸本)9781538636497

Recently, the amount of array data generated by scientific observation instruments increases rapidly. The array data is usually stored in standard formats such as HDF5 and NetCDF. To support high-level queries on the array data, a number of array DBMSs such as SciDB have been proposed. However, they typically have two drawbacks: slow data loading and not directly supporting standard formats. In particular, slow data loading is fatal since the speed of scientific data generation might be faster than that of data loading. To solve those drawbacks, we propose a distributed in-situ processing system called SciDFS that exploits a distributed file system (DFS) for storing and managing array data. SciDFS is a hybrid system that tightly integrates the query processing layer of an array DBMS with a DFS via an in-situ layer. It stores raw array data as DFS blocks very fast and processes queries in an in-situ manner by accessing the relevant DFS blocks. Through experiments using NASA's real satellite array data, we have shown three major features of SciDFS: high performance data loading (50X faster than SciDB), fast in-situ query processing performance, and running legacy applications for the HDF5 format.

关键词： in-situ processing distributed file system array DBMS big data analysis

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：