检索结果-内蒙古大学图书馆

YZStack: Provisioning Customizable Solution for Big Data

PROCEEDINGS OF THE VLDB ENDOWMENT 2014年第13期7卷 1778-1783页

作者： Wu, Sai Chen, Chun Chen, Gang Chen, Ke Shou, Lidan Cao, Hui Bai, He Zhejiang Univ Coll Comp Sci & Technol Hangzhou Zhejiang Peoples R China YzBigData Co Ltd Room 205Tower BHangzhou Narada Grand Hotel Hangzhou Zhejiang Peoples R China City Cloud Technol Hangzhou Co Ltd Hangzhou Zhejiang Peoples R China

YZStack is our developing solution which implements many wellestablished big data techniques as selectable modules and allows users to customize their systems as a process of module selection. In particular, it includes an openstack based IaaS (Infrastructure as a Service) layer, a distributed file system based DaaS (Data as a Service) layer, a PaaS (Platform as a Service) layer equipped wit parallel processing techniques and a SaaS (Software as a Service) layer with popular data analytic algorithms. Layers of YZStack are loosely connected, so that customization of one layer does not affect the other layers and their interactions. In this paper, we use a smart financial system developed for the Zhejiang Provincial Department of Finance to demonstrate how to leverage YZStack to speed up the implementation of big data system. We also introduce two popular applications of the financial system, economic prediction and detection of improper payment.

关键词： Big data Data as a service (DaaS) file organization Finance Infrastructure as a service (IaaS) Multiprocessing systems Platform as a Service (PaaS) Software as a service (SaaS) Analytic algorithm Big data systems Developing solutions distributed file systems Financial system Module selection Parallel processing SaaS (software as a service)

来源：评论

学校读者我要写书评

暂无评论

Extensible block-level storage virtualization in cluster-based systems

引用

JOURNAL OF PARALLEL AND distributed COMPUTING 2010年第8期70卷 800-824页

作者： Flouris, Michail D. Lachaize, Renaud Chasapis, Konstantinos Bilas, Angelos Fdn Res & Technol Hellas Inst Comp Sci GR-70013 Iraklion Greece Univ Crete Dept Comp Sci GR-71409 Iraklion Greece

High-performance storage systems are evolving towards decentralized commodity clusters that can scale in capacity, processing power, and network throughput. Building such systems requires: (a) Sharing physical resources among applications;(b) Sharing data among applications;(c) Allowing customized data views. Current solutions typically satisfy the first two requirements through a cluster file-system, resulting in monolithic, hard-to-manage systems. In this paper we present a storage system that addresses all three requirements by extending the block layer below the file-system. First, we discuss how our system provides customized (virtualized) storage views within a single node. Then, we discuss how it scales in clustered setups. To achieve efficient resource and data sharing we support block-level allocation and locking as in-band mechanisms. We implement a prototype under Linux and use it to build a shared cluster file-system. Our evaluation in a 24-node cluster setup concludes that our approach offers flexibility, scalability and reduced effort to implement new functionality. (C) 2010 Elsevier Inc. All rights reserved.

关键词： Cluster storage Scalable storage systems Block-level I/O Storage virtualization Extensible storage stack Shared virtual disk distributed file systems

来源：评论

学校读者我要写书评

暂无评论

Metadata Distribution and Consistency Techniques for Large-Scale Cluster file systems

引用

IEEE TRANSACTIONS ON PARALLEL AND distributed systems 2011年第5期22卷 803-816页

作者： Xiong, Jin Hu, Yiming Li, Guojie Tang, Rongfeng Fan, Zhihua Chinese Acad Sci Inst Comp Technol Beijing 100190 Peoples R China Univ Cincinnati Dept Elect & Comp Engn & Comp Sci Engn Res Ctr 542 Cincinnati OH 45221 USA NetEase Com Inc Beijing 100084 Peoples R China

Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads.

关键词： distributed file systems metadata management

来源：评论

学校读者我要写书评

暂无评论

The big data system, components, tools, and technologies: a survey

引用

KNOWLEDGE AND INFORMATION systems 2019年第3期60卷 1165-1245页

作者： Rao, T. Ramalingeswara Mitra, Pabitra Bhatt, Ravindara Goswami, A. Indian Inst Technol Kharagpur Dept Math Theoret Comp Sci Grp Kharagpur 721302 W Bengal India Indian Inst Technol Kharagpur Dept Comp Sci & Engn Kharagpur 721302 W Bengal India Jaypee Univ Informat Technol Dept Comp Sci & Engn Waknaghat 173234 India

The traditional databases are not capable of handling unstructured data and high volumes of real-time datasets. Diverse datasets are unstructured lead to big data, and it is laborious to store, manage, process, analyze, visualize, and extract the useful insights from these datasets using traditional database approaches. However, many technical aspects exist in refining large heterogeneous datasets in the trend of big data. This paper aims to present a generalized view of complete big data system which includes several stages and key components of each stage in processing the big data. In particular, we compare and contrast various distributed file systems and MapReduce-supported NoSQL databases concerning certain parameters in data management process. Further, we present distinct distributed/cloud-based machine learning (ML) tools that play a key role to design, develop and deploy data models. The paper investigates case studies on distributed ML tools such as Mahout, Spark MLlib, and FlinkML. Further, we classify analytics based on the type of data, domain, and application. We distinguish various visualization tools pertaining three parameters: functionality, analysis capabilities, and supported development environment. Furthermore, we systematically investigate big data tools and technologies (Hadoop 3.0, Spark 2.3) including distributed/cloud-based stream processing tools in a comparative approach. Moreover, we discuss functionalities of several SQL Query tools on Hadoop based on 10 parameters. Finally, We present some critical points relevant to research directions and opportunities according to the current trend of big data. Investigating infrastructure tools for big data with recent developments provides a better understanding that how different tools and technologies apply to solve real-life applications.

关键词： Big data Components of big data system distributed file systems NoSQL databases Visualization SQL Query tools Data analytics

来源：评论

学校读者我要写书评

暂无评论

distributed OPERATING-systems

引用

COMPUTER STANDARDS & INTERFACES 1987年第1期6卷 37-44页

作者： MULLENDER, SJ CTR MATH & COMP SCI DISTRIBUTED SYST GRP1098 SJ AMSTERDAMNETHERLANDS

In the past five years, distributed operating systems research has gone through a consolidation phase. On a large number of design issues there is now considerable consensus between different research groups. In this paper, an overview of recent research in distributed systems is given. In turn, the paper discusses overall system structure, protection issues, file system designs, problems and solutions for fault tolerance and a mechanism that is rapidly becoming very important for efficient distributed systems design: hints. An attempt was made to provide sufficient references to interesting research projects for the reader to find material for more detailed study.

关键词： distributed operating systems distributed file systems fault tolerance

来源：评论

学校读者我要写书评

暂无评论

FICUS - A federated service-oriented file transfer framework

FICUS - A federated service-oriented file transfer framework

引用

14th International Conference on Concurrent Engineering

作者： Turner, Adam Sobolewski, Michael Texas Tech Univ SORCER Res Grp Lubbock TX USA

ISBN: (纸本)9781846289750

The engineering data of a large enterprise is typically distributed over a wide area and archived in a variety of file systems and databases. Access to such information is crucial to team members and relevant processing services (applications, tools and utilities) in a concurrent engineering setting. However, this is not easy because there is no simple way to efficiently access the information without being knowledgeable about various file systems, file servers, and networks, especially when complex domain related data files get bigger. In a concurrent engineering environment, there is every need to be aware of the transparent and dynamic data perspectives of the other members of the team. We have developed a Federated Service-Oriented file Transfer Framework called FICUS (files In Chunks Utilizing Storage) with the objective to form dynamic federations of network services that provide engineering data, applications and tools on a grid. This framework fits the SORCER philosophy of grid interactive service-oriented programming, where users create distributed metaprograms using federated providers along with FICUS repository providers. Our paper describes the methodology of how FICUS works along with the details of the implementation and extensions planned for the future. We believe the performance and reliability offered by FICUS will make it a very useful distributed file transfer protocol for a large design team and will make it very convenient to integrate heterogeneous legacy file systems.

关键词： data sharing distributed file systems federated systems collaborative work

来源：评论

学校读者我要写书评

暂无评论

Towards seamless mobility on pervasive hardware

引用

PERVASIVE AND MOBILE COMPUTING 2005年第2期1卷 157-189页

作者： Satyanarayanan, M. Kozuch, Michael A. Helfrich, Casey J. O'Hallaron, David R. Carnegie Mellon Univ Sch Comp Sci 5000 Forbes Ave Pittsburgh PA 15213 USA Intel Res Pittsburgh Pittsburgh PA USA

Preserving one's uniquely customized computing environment as one moves to different locations is an enduring challenge in mobile computing. We examine why this capability is valued so highly, and what makes it so difficult to achieve for personal computing applications. We describe a new mechanism called Internet Suspend/Resume (ISR) that overcomes many of the limitations of previous approaches to realizing this capability. ISR enables a hands-free approach to mobile computing that appears well suited to future pervasive computing environments in which commodity hardware may be widely deployed for transient use. We show that ISR can be implemented by layering virtual machine technology on distributed file system technology. We also report on measurements from a prototype that confirm that ISR is already usable today for some common usage scenarios. (C) 2005 Elsevier B.V. All rights reserved.

关键词： Mobile computing Pervasive computing Personal computing Internet Suspend/Resume Virtual machines VMware distributed file systems Coda Portable storage Seamless mobility Human attention Thin clients Process migration

来源：评论

学校读者我要写书评

暂无评论

Toward Efficient Block Replication Management in distributed Storage

引用

ACM TRANSACTIONS ON MODELING AND PERFORMANCE EVALUATION OF COMPUTING systems 2020年第3期5卷 1–27页

作者： Liao, Jianwei Sha, Zhibing Cai, Zhigang Liu, Zhiming Li, Kenli Liao, Wei-Keng Choudhary, Alok N. Ishiakwa, Yutaka Southwest Univ Chongqing Peoples R China Hunan Univ Changsha Hunan Peoples R China Northwestern Univ Evanston IL 60208 USA RIKEN Tokyo Japan

distributed/parallel file systems commonly suffer from load imbalance and resource contention due to the bursty characteristic exhibited in scientific applications. This article presents an adaptive scheme supporting dynamic block data replication and an efficient replica placement policy to improve the I/O performance of a distributed file system. Our goal is not only to yield a balanced data replication among storage servers but also a high degree of data access parallelism for the applications. We first present mathematical cost models to formulate the cost of data block replication by considering both the overhead and reduced data access time to the replicated data. To verify the validity and feasibility of the proposed cost model, we implement our proposal in a prototype distributed file system and evaluate it using a set of representative database-relevant application benchmarks. Our results demonstrate that the proposed approach can boost the usage efficiency of the data replicas with acceptable overhead of data replication management. Consequently, the overall data throughput of storage system can be noticeably improved. In summary, the proposed replication management scheme works well, especially for the database-relevant applications that exhibit an uneven access frequency and pattern to different parts of files.

关键词： distributed file systems block data replication modeling replica placement access load balance

来源：评论

学校读者我要写书评

暂无评论

Adaptable SLA-Aware Consistency Tuning for Quorum-Replicated Datastores

引用

IEEE TRANSACTIONS ON BIG DATA 2017年第3期3卷 248-261页

作者： Sidhanta, Subhajit Golab, Wojciech Mukhopadhyay, Supratik Basu, Saikat Univ Lisbon INESC ID IST P-1600276 Lisbon Portugal Louisiana State Univ Baton Rouge LA 70803 USA Louisiana State Univ Dept Comp Sci Baton Rouge LA 70803 USA Univ Waterloo Elect & Comp Engn Waterloo ON N2L 3G1 Canada

Users of distributed datastores that employ quorum- based replication are burdened with the choice of a suitable client-centric consistency setting for each storage operation. The above matching choice is difficult to reason about as it requires deliberating about the tradeoff between the latency and staleness, i.e., how stale (old) the result is. The latency and staleness for a given operation depend on the client-centric consistency setting applied, as well as dynamic parameters such as the current workload and network condition. We present OptCon, a machine learning-based predictive framework, that can automate the choice of client-centric consistency setting under user-specified latency and staleness thresholds given in the service level agreement (SLA). Under a given SLA, OptCon predicts a client-centric consistency setting that is matching, i.e., it is weak enough to satisfy the latency threshold, while being strong enough to satisfy the staleness threshold. While manually tuned consistency settings remain fixed unless explicitly reconfigured, OptCon tunes consistency settings on a per-operation basis with respect to changing workload and network state. Using decision tree learning, OptCon yields 0.14 cross validation error in predicting matching consistency settings under latency and staleness thresholds given in the SLA. We demonstrate experimentally that OptCon is at least as effective as any manually chosen consistency settings in adapting to the SLA thresholds for different use cases. We also demonstrate that OptCon adapts to variations in workload, whereas a given manually chosen fixed consistency setting satisfies the SLA only for a characteristic workload.

关键词： distributed databases storage management distributed systems storage management high availability distributed file systems middleware performance evaluation machine learning reliability availability serviceability

来源：评论

学校读者我要写书评

暂无评论

BlockIPFS - Blockchain-enabled Interplanetary file System for Forensic and Trusted Data Traceability 2

BlockIPFS - Blockchain-enabled Interplanetary File System fo...

引用

2nd IEEE International Conference on Blockchain (Blockchain)

作者： Nyaletey, Emmanuel Parizi, Reza M. Zhang, Qi Choo, Kim-Kwang Raymond Kennesaw State Univ Coll Comp & Software Engn Kennesaw GA 30144 USA IBM Thomas J Watson Res Yorktown Hts NY USA Univ Texas San Antonio Dept Informat Syst & Cyber Secur San Antonio TX USA

ISBN: (纸本)9781728146935

The Interplanetary file System (IPFS) is a distributed file system that seeks to decentralize the web and to make it faster and more efficient. It incorporates well-known technologies, including BitTorrent and Git, to create a swarm of computing systems that share information. Since its introduction in 2016, IPFS has seen great improvements and adoption from both individuals and enterprise organizations. Its distributed network allows users to share files and information across the globe. IPFS works well with large files that may consume or require large bandwidth to upload and/or download over the Internet. The rapid adoption of this distributed file system is in part because IPFS is designed to operate on top of different protocols, such as FTP and HTTP. However, there are underpinning concerns relating to security and access control, for example lack of traceability on how the files are accessed. The aim of this paper is to complement IPFS with blockchain technology, by proposing a new approach (BlockIPFS) to create a clear audit trail. BlockIPFS allows us to achieve improved trustworthiness of the data and authorship protection, and provide a clear route to trace back all activities associated with a given file using blockchain as a service.

关键词： Blockchains Blockchain as a Service Smart Contracts Hyperledger Fabric IPFS distributed file systems

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：