检索结果-内蒙古大学图书馆

A balanced scheduler with data reuse and replication for scientific workflows in cloud computing systems

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2017年 74卷 168-178页

作者： Casas, Israel Taheri, Javid Ranjan, Rajiv Wang, Lizhe Zomaya, Albert Y. Univ Sydney Sch Informat Technol Sydney NSW Australia Newcastle Univ Sch Comp Sci Newcastle Upon Tyne Tyne & Wear England CSIRO Data61 Canberra ACT Australia China Univ Geosci Sch Comp Sci Wuhan 430074 Peoples R China Karlstad Univ Dept Math & Comp Sci Karlstad Sweden

Cloud computing provides substantial opportunities to researchers who demand pay-as-you-go computing systems. Although cloud provider (e.g., Amazon Web Services) and application provider (e.g., biologists, physicists, and online gaming companies) both have specific performance requirements (e.g. application response time), it is the cloud scheduler's responsibility to map the application to underlying cloud resources. This article presents a Balanced and file Reuse Replication Scheduling (BaRRS) algorithm for cloud computing environments to optimally schedule scientific application workflows. BaRRS splits scientific workflows into multiple sub-workflows to balance system utilization via parallelization. It also exploits data reuse and replication techniques to optimize the amount of data that needs to be transferred among tasks at run-time. BaRRS analyzes the key application features (e.g., task execution times, dependency patterns and file sizes) of scientific workflows for adapting existing data reuse and replication techniques to cloud systems. Further, BaRRS performs a trade-off analysis to select the optimal solution based on two optimization constraints: execution time and monetary cost of running scientific workflows. BaRRS is compared with a state-of-the-art scheduling approach;experiments prove its superior performance. Experiments include four well known scientific workflows with different dependency patterns and data file sizes. Results were promising and also highlighted most critical factors affecting execution of scientific applications on clouds. (C) 2015 Elsevier B.V. All rights reserved.

关键词： Cloud computing Scientific workflow Scheduling Virtual machine data-intensive computing Big data

来源：评论

学校读者我要写书评

暂无评论

ScaDS Dresden/Leipzig - A competence center for collaborative big data research

引用

IT-INFORMATION TECHNOLOGY 2018年第5-6期60卷 327-333页

作者： Jaekel, Rene Peukert, Eric Nagel, Wolfgang E. Rahm, Erhard Tech Univ Dresden Ctr Informat Serv & High Performance Comp ZIH D-01062 Dresden Germany Big Data Kompetenzzentrum ScaDS Dresden Leipzig Ritterstr 9-132OG D-04109 Leipzig Germany Univ Leipzig Fak Math & Informat Augustuspl 10 D-04109 Leipzig Germany

The efficient and intelligent handling of large, often distributed and heterogeneous data sets increasingly determines the scientific and economic competitiveness in most application areas. Mobile applications, social networks, multimedia collections, sensor networks, data intense scientific experiments, and complex simulations nowadays generate a huge data deluge. Nonetheless, processing and analyzing these data sets with innovative methods open up new opportunities for its exploitation and new insights. Nevertheless, the resulting resource requirements exceed usually the possibilities of state-of-the-art methods for the acquisition, integration, analysis and visualization of data and are summarized under the term big data. ScaDS Dresden/Leipzig, as one Germany-wide competence center for collaborative big data research, bundles efforts to realize data-intensive applications for a wide range of applications in science and industry. In this article, we present the basic concept of the competence center and give insights in some of its research topics.

关键词： big data data-intensive computing data integration collaborative research knowledge extraction visualization high-performance computing

来源：评论

学校读者我要写书评

暂无评论

Enabling Strategies for Big data Analytics in Hybrid Infrastructures 16

Enabling Strategies for Big Data Analytics in Hybrid Infrast...

引用

International Conference on High Performance computing & Simulation (HPCS)

作者： Anjos, Julio C. S. Matteussi, Kassiano J. De Souza, Paulo R. R. Geyer, Claudio F. R. Veith, Alexandre S. Fedak, Gilles Victoria Barbosa, Jorge Luis Univ Fed Rio Grande do Sul Inst Informat BR-91509900 Porto Alegre RS Brazil ENS Lyon INRIA LIP Lyon France Univ Vale Rio dos Sinos Appl Comp Grad Program Sao Leopoldo Brazil

ISBN: (纸本)9781538678794

A huge volume of data is produced every day by social networks (e.g. Facebook, Instagram, Whatsapp, etc.), sensors, mobile devices and other applications. Although the Cloud computing scenario has grown rapidly in recent years, it still suffers from a lack of the kind of standardization that involves the resource management for Big data applications, such as the case of MapReduce. In this context, the users face a big challenge in attempting to understand the requirements of the application and how to consolidate the resources properly. This scenario raises significant challenges in the different areas: systems, infrastructure, platforms as well as providing several research opportunities in Big data Analytics. This work proposes the use of hybrid infrastructures such as Cloud and Volunteer computing for Big data processing and analysis. In addition, it provides a data distribution model that improves the resource management of Big data applications in hybrid infrastructures. The results indicate the feasibility of hybrid infrastructures since it supports the reproducibility and predictability of Big data processing by low and high-scale simulation within Hybrid infrastructures.

关键词： Big data MapReduce Hybrid Infrastructures Distributed System data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

A capabilities-aware framework for using computational accelerators in data-intensive computing

引用

JOURNAL OF PARALLEL AND DISTRIBUTED computing 2011年第2期71卷 185-197页

作者： Rafique, M. Mustafa Butt, Ali R. Nikolopoulos, Dimitrios S. Virginia Tech Dept Comp Sci Blacksburg VA 24061 USA FORTH Inst Comp Sci GR-70013 Iraklion Greece Univ Crete Dept Comp Sci GR-71409 Iraklion Greece

Multicore computational accelerators such as GPUs are now commodity components for high-performance computing at scale. While such accelerators have been studied in some detail as stand-alone computational engines, their integration in large-scale distributed systems raises new challenges and trade-offs. In this paper, we present an exploration of resource management alternatives for building asymmetric accelerator-based distributed systems. We present these alternatives in the context of a capabilities-aware framework for data-intensive computing, which uses an enhanced implementation of the MapReduce programming model for accelerator-based clusters, compared to the state of the art. The framework can transparently utilize heterogeneous accelerators for deriving high performance with low programming effort. Our work is the first to compare heterogeneous types of accelerators, GPUs and a Cell processors, in the same environment and the first to explore the trade-offs between compute-efficient and control-efficient accelerators on data-intensive systems. Our investigation shows that our framework scales well with the number of different compute nodes. Furthermore, it runs simultaneously on two different types of accelerators, successfully adapts to the resource capabilities, and performs 26.9% better on average than a static execution approach. (C) 2010 Elsevier Inc. All rights reserved.

关键词： Cloud computing MapReduce Asymmetric multicores Programming accelerators data-intensive computing Heterogeneous clusters

来源：评论

学校读者我要写书评

暂无评论

Passive Network Performance Estimation for Large-Scale, data-intensive computing

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2011年第8期22卷 1365-1373页

作者： Kim, Jinoh Chandra, Abhishek Weissman, Jon B. Univ Minnesota Dept Comp Sci & Engn Minneapolis MN 55455 USA

Distributed computing applications are increasingly utilizing distributed data sources. However, the unpredictable cost of data access in large-scale computing infrastructures can lead to severe performance bottlenecks. Providing predictability in data access is, thus, essential to accommodate the large set of newly emerging large-scale, data-intensive computing applications. In this regard, accurate estimation of network performance is crucial to meeting the performance goals of such applications. Passive estimation based on past measurements is attractive for its relatively small overhead compared to relying on explicit probing. In this paper, we take a passive approach for network performance estimation. Our approach is different from existing passive techniques that rely either on past direct measurements of pairs of nodes or on topological similarities. Instead, we exploit secondhand measurements collected by other nodes without any topological restrictions. In this paper, we present Overlay Passive Estimation of Network performance (OPEN), a scalable framework providing end-to-end network performance estimation based on secondhand measurements, and discuss how OPEN achieves cost-effective estimation in a large-scale infrastructure. Our extensive experimental results show that OPEN estimation can be applicable for replica and resource selections commonly used in distributed computing.

关键词： Network performance estimation secondhand estimation data-intensive computing replica selection resource selection

来源：评论

学校读者我要写书评

暂无评论

Optimizing end-to-end performance of data-intensive computing pipelines in heterogeneous network environments

引用

JOURNAL OF PARALLEL AND DISTRIBUTED computing 2011年第2期71卷 254-265页

作者： Wu, Qishi Gu, Yi Univ Memphis Dept Comp Sci Memphis TN 38152 USA

Supporting high-performance data-intensive computing pipelines in wide-area networks is crucial for enabling large-scale distributed scientific applications that require minimizing end-to-end delay for single-input applications or maximizing frame rate for streaming applications. We formulate and categorize the data-intensive computing pipeline mapping problems into six classes with two optimization objectives, i.e. minimum end-to-end delay and maximum frame rate, and three network constraints, i.e. no, contiguous, and arbitrary node reuse. We design a dynamic programming-based optimal solution to the problem of minimum end-to-end delay with arbitrary node reuse and prove the NP-completeness of the rest five problems, for each of which, a heuristic algorithm based on a similar optimization procedure is proposed. These heuristics are implemented and tested on a large set of simulated pipelines and networks of various scales and their performance superiority is illustrated by extensive simulation results in comparison with existing methods. (C) 2010 Elsevier Inc. All rights reserved.

关键词： data-intensive computing Pipeline Performance optimization End-to-end delay Frame rate

来源：评论

学校读者我要写书评

暂无评论

Scalable Metadata Management Techniques for Ultra-Large Distributed Storage Systems - A Systematic Review

引用

ACM computing SURVEYS 2018年第4期51卷 82-82页

作者： Singh, Harcharan Jit Bawa, Seema Thapar Univ Comp Sci & Engn Dept Patiala 147004 Punjab India

The provisioning of an efficient ultra-large scalable distributed storage system for expanding cloud applications has been a challenging job for researchers in academia and industry. In such an ultra-large-scale storage system, data are distributed onmultiple storage nodes for performance, scalability, and availability. The access to this distributed data is through itsmetadata, maintained bymultiple metadata servers. Themetadata carries information about the physical address of data and access privileges. The efficiency of a storage system highly depends on effective metadata management. This research presents an extensive systematic literature analysis of metadata management techniques in storage systems. This research work will help researchers to find the significance of metadata management and important parameters of metadata management techniques for storage systems. Methodical examination of metadata management techniques developed by various industry and research groups is described. The different metadata distribution techniques lead to various taxonomies. Furthermore, the article investigates techniques based on distribution structures and key parameters of metadata management. It also presents strengths and weaknesses of individual existing techniques that will help researchers to select the most appropriate technique for specific applications. Finally, it discusses existing challenges and significant research directions in metadata management for researchers.

关键词： Distributed computing big data storage systems distributed file system data-intensive computing metadata management scalability load balancing locality namespace

来源：评论

学校读者我要写书评

暂无评论

Accelerating big data analytics on HPC clusters using two-level storage

引用

PARALLEL computing 2017年 61卷 18-34页

作者： Xuan, Pengfei Ligon, Walter B. Srimani, Pradip K. Ge, Rong Luo, Feng Clemson Univ Sch Comp Clemson SC 29634 USA Clemson Univ Elect & Comp Engn Clemson SC 29634 USA

data-intensive applications that are inherently I/O bound have become a major workload on traditional high-performance computing (HPC) clusters. Simply employing data intensive computing storage such as HDFS or using parallel file systems available on HPC clusters to serve such applications incurs performance and scalability issues. In this paper, we present a novel two-level storage system that integrates an upper-level in-memory file system with a lower-level parallel file system. The former renders memory-speed high I/O performance and the latter renders consistent storage with large capacity. We build a two level storage system prototype with Tachyon and OrangeFS, and analyze the resulting I/O throughput for typical MapReduce operations. Theoretical modeling and experiments show that the proposed two-level storage delivers higher aggregate I/O throughput than HDFS and OrangeFS and achieves scalable performance for both read and write. We expect this two-level storage approach to provide insights on system design for big data analytics on HPC clusters. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Two-level storage In-memory file system Parallel file system data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

Large-scale biological meta-database management

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2017年 67卷 481-489页

作者： Pedersen, Edvard Bongo, Lars Ailo Univ Tromso Dept Comp Sci Tromso Norway Univ Tromso Ctr Bioinformat Tromso Norway

Up-to-date meta-databases are vital for the analysis of biological data. However, the current exponential increase in biological data leads to exponentially increasing meta-database sizes. Large-scale meta database management is therefore an important challenge for production platforms providing services for biological data analysis. In particular, there is often a need either to run an analysis with a particular version of a meta-database, or to rerun an analysis with an updated meta-database. We present our GeStore approach for biological meta-database management. It provides efficient storage and runtime generation of specific meta-database versions, and efficient incremental updates for biological data analysis tools. The approach is transparent to the tools, and we provide a framework that makes it easy to integrate GeStore with biological data analysis frameworks. We present the GeStore system, an evaluation of the performance characteristics of the system, and an evaluation of the benefits for a biological data analysis workflow. (C) 2016 Elsevier B.V. All rights reserved.

关键词： Bioinformatics Big data management Hadoop data-intensive computing Metagenomics

来源：评论

学校读者我要写书评

暂无评论

Hyracks: A Flexible and Extensible Foundation for data-intensive computing

Hyracks: A Flexible and Extensible Foundation for Data-Inten...

引用

IEEE 27th International Conference on data Engineering (ICDE 2011)

作者： Borkar, Vinayak Carey, Michael Grover, Raman Onose, Nicola Vernica, Rares Univ Calif Irvine Dept Comp Sci Irvine CA 92697 USA

ISBN: (纸本)9781424489589

Hyracks is a new partitioned-parallel software platform designed to run data-intensive computations on large shared-nothing clusters of computers. Hyracks allows users to express a computation as a DAG of data operators and connectors. Operators operate on partitions of input data and produce partitions of output data, while connectors repartition operators' outputs to make the newly produced partitions available at the consuming operators. We describe the Hyracks end user model, for authors of dataflow jobs, and the extension model for users who wish to augment Hyracks' built-in library with new operator and/or connector types. We also describe our initial Hyracks implementation. Since Hyracks is in roughly the same space as the open source Hadoop platform, we compare Hyracks with Hadoop experimentally for several different kinds of use cases. The initial results demonstrate that Hyracks has significant promise as a next-generation platform for data-intensive applications.

关键词： Computational modeling Computer architecture DAG Hyracks end user model data connectors data handling data operators data-intensive computing open source Hadoop platform parallel programming partitioned-parallel software platform

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：