检索结果-内蒙古大学图书馆

IEEE International Conference on Big data (Big data)

作者： Liu, Jialin Crysler, Bradly Lu, Yin Chen, Yong Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

ISBN: (纸本)9781479912926;9781479912933

Scientific I/O libraries, like PnetCDF, ADIOS, and HDF5, have been commonly used to facilitate the array-based scientific dataset processing. The underlying physical data layout information, however, is usually hidden from the upper layer's logical access. Such mismatching can lead to poor I/O. In this research, we have observed performance degradation in the case of concurrent sub-array accesses, where overlaps among calls that access sub-arrays led to high contention on storage servers due to the logical-physical mismatching. We propose a locality-driven high-level I/O aggregation approach to address these issues in this work. By designing a logical-physical mapping scheme, we try to utilize the scientific dataset's structured formats and the file systems' data distribution to resolve the mismatching issue. Therefore the I/O can be carried out in a locality-driven fashion. The proposed approach is effective and complements the existing I/O strategies, such as the independent I/O and collective I/O strategy. We have also carried out experimental tests and the results confirm the performance improvement compared to existing I/O strategies. The proposed locality-driven high-level I/O aggregation approach holds a promise for efficiently processing scientific datasets, which is critical for the data intensive or big data computing era.

关键词： Big data data intensive computing high performance computing collective I/O I/O aggregation scientific I/O library

来源：评论

学校读者我要写书评

暂无评论

MapReduce Workload Modeling with Statistical Approach

引用

JOURNAL OF GRID computing 2012年第2期10卷 279-310页

作者： Yang, Hailong Luan, Zhongzhi Li, Wenjun Qian, Depei Beihang Univ Sino German Joint Software Inst State Key Lab Software Dev Environm Sch Comp Sci & Engn Beijing Peoples R China

Large-scale data-intensive cloud computing with the MapReduce framework is becoming pervasive for the core business of many academic, government, and industrial organizations. Hadoop, a state-of-the-art open source project, is by far the most successful realization of MapReduce framework. While MapReduce is easy- to-use, efficient and reliable for data-intensive computations, the excessive configuration parameters in Hadoop impose unexpected challenges on running various workloads with a Hadoop cluster effectively. Consequently, developers who have less experience with the Hadoop configuration system may devote a significant effort to write an application with poor performance, either because they have no idea how these configurations would influence the performance, or because they are not even aware that these configurations exist. There is a pressing need for comprehensive analysis and performance modeling to ease MapReduce application development and guide performance optimization under different Hadoop configurations. In this paper, we propose a statistical analysis approach to identify the relationships among workload characteristics, Hadoop configurations and workload performance. We apply principal component analysis and cluster analysis to 45 different metrics, which derive relationships between workload characteristics and corresponding performance under different Hadoop configurations. Regression models are also constructed that attempt to predict the performance of various workloads under different Hadoop configurations. Several non-intuitive relationships between workload characteristics and performance are revealed through our analysis and the experimental results demonstrate that our regression models accurately predict the performance of MapReduce workloads under different Hadoop configurations.

关键词： Cloud computing data intensive computing MapReduce Workload characterization Statistical analysis Performance prediction

来源：评论

学校读者我要写书评

暂无评论

Analyzing design choices for distributed multidimensional indexing

引用

JOURNAL OF SUPERcomputing 2012年第3期59卷 1552-1576页

作者： Nam, Beomseok Sussman, Alan Ulsan Natl Inst Sci & Technol Sch Elect & Comp Engn Ulsan 689798 South Korea Univ Maryland Dept Comp Sci College Pk MD 20742 USA

Scientific datasets are often stored on distributed archival storage systems, because geographically distributed sensor devices store the datasets in their local machines and also because the size of scientific datasets demands large amount of disk space. Multidimensional indexing techniques have been shown to greatly improve range query performance into large scientific datasets. In this paper, we discuss several ways of distributing a multidimensional index in order to speed up access to large distributed scientific datasets. This paper compares the designs, challenges, and problems for distributed multidimensional indexing schemes, and provides a comprehensive performance study of distributed indexing to provide guidelines to choose a distributed multidimensional index for a specific data analysis application.

关键词： Multidimensional indexing Distributed indexing Decentralized indexing data intensive computing

来源：评论

学校读者我要写书评

暂无评论

Statistics-based Workload Modeling for MapReduce

Statistics-based Workload Modeling for MapReduce

引用

26th IEEE International Parallel and Distributed Processing Symposium (IPDPS) / Workshop on High Performance data intensive computing

作者： Yang, Hailong Luan, Zhongzhi Li, Wenjun Qian, Depei Guan, Gang Beihang Univ Sch Comp Sci Beijing Peoples R China Tencent Res Beijing Peoples R China

ISBN: (纸本)9780769546766

Large-scale data-intensive computing with MapReduce framework in Cloud is becoming pervasive for the core business of many academic, government, and industrial organizations. Hadoop is by far the most successful realization of MapReduce framework. While MapReduce is easy-to-use, efficient and reliable for data-intensive computations, the excessive configuration parameters in Hadoop cause unexpected challenges when running various workloads with Hadoop cluster effectively. Consequently, developers who have less experience with the Hadoop configuration system may devote a significant effort to write an application with poor performance, because they have no idea how these configurations would influence the performance, or they are not even aware that these configurations exist. In this paper, we propose a statistic analysis approach to identify the relationships among workload characteristics, Hadoop configurations and workload performance. Several non-intuitive relationships between workload characteristics and relative performance are revealed and the experimental results demonstrate that our regression models accurately predict the performance of MapReduce workloads under different Hadoop configurations.

关键词： data intensive computing MapReduce analytical model performance prediction

来源：评论

学校读者我要写书评

暂无评论

Improving data Analysis Performance for High-Performance computing with Integrating Statistical Metadata in Scientific datasets

Improving Data Analysis Performance for High-Performance Com...

引用

25th ACM/IEEE International Conference for High Performance computing, Networking, Storage and Analysis (SC)

作者： Liu, Jialin Chen, Yong Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

ISBN: (纸本)9780769549569;9781467362184

Scientific datasets and libraries, such as HDF5, ADIOS, and NetCDF, have been used widely in many data intensive applications. These libraries have their special file formats and I/O functions to provide efficient access to large datasets. When the data size keeps increasing, these high level I/O libraries face new challenges. Recent studies have started to utilize database techniques such as indexing and subsetting, and data reorganization to manage the increasing datasets. In this work, we present a new approach to boost the data analysis performance, namely Fast Analysis with Statistical Metadata (FASM), via data subsetting and integrating a small amount of statistics into the original datasets. The added statistical information illustrates the data shape and provides knowledge of the data distribution;therefore the original I/O libraries can utilize these statistical metadata to perform fast queries and analyses. The proposed FASM approach is currently evaluated with the PnetCDF on Lustre file systems, but can also be implemented with other scientific libraries. The FASM can potentially lead to a new dataset design and can have an impact on big data analysis.

关键词： high performance computing data intensive computing big data storage systems statistical techniques FASM

来源：评论

学校读者我要写书评

暂无评论

OpenFlow Enabled Hadoop Over Local and Wide Area Clusters

OpenFlow Enabled Hadoop Over Local and Wide Area Clusters

引用

25th ACM/IEEE International Conference for High Performance computing, Networking, Storage and Analysis (SC)

作者： Narayan, Sandhya Bailey, Stuart Greenway, Matthew Grossman, Robert Heath, Allison Powell, Ray Daga, Anand InfoBlox Inc Santa Clara CA USA Univ Chicago Inst Comp Chicago IL USA Univ Houston Dept Comp Sci Houston TX USA

ISBN: (纸本)9780769549569;9781467362184

Hadoop has emerged as an important platform for data intensive computing. The shuffle and sort phases of a MapReduce computation often saturate top of the rack switches, as well as switches that aggregate multiple racks. In addition, MapReduce computations often have "hot spots" in which the computation is lengthened due to inadequate bandwidth to some of the nodes. In principle, OpenFlow enables an application to adjust the network topology as required by the computation, providing additional network bandwidth to those resources requiring it. We describe Hadoop-OFE, which is an OpenFlow enabled version of Hadoop that dynamically modifies the network topology in order to improve the performance of Hadoop.

关键词： Hadoop OpenFlow OpenFlow over Ethernet data intensive computing MapReduce

来源：评论

学校读者我要写书评

暂无评论

DOSAS: Mitigating the Resource Contention in Active Storage Systems

DOSAS: Mitigating the Resource Contention in Active Storage ...

引用

IEEE International Conference on Cluster computing

作者： Chen, Chao Chen, Yong Roth, Philip C. Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA Oak Ridge Natl Lab Comp Sci & Math Div Oak Ridge TN USA

ISBN: (纸本)9781467324229

Active storage provides an effective method to mitigate the I/O bottleneck problem of data intensive high performance computing applications. It can reduce the amount of data transferred as the application runs by moving appropriate computations close to the data. Prior research has achieved considerable progress in developing several active storage prototypes. However, existing studies have neglected the impact of resource contention when concurrent processes request IO operations from the same storage node simultaneously, which happens frequently in practice. In this paper, we analyze the impact of resource contention on active storage systems. Motivated by our analysis, we propose a novel Dynamic Operation Scheduling Active Storage architecture to address the resource contention issue. It offloads the active processing operations dynamically between storage nodes and compute nodes according to the system environment. By evaluating our architecture, we observed that: (1) resource contention is a critical problem for active storage systems, (2) the proposed dynamic operation scheduling method mitigates the problem, and (3) the new active storage architecture outperforms existing active storage systems.

关键词： dynamic active storage active storage resource contention high performance computing data intensive computing parallel I/O parallel file systems

来源：评论

学校读者我要写书评

暂无评论

MapReduce Across Distributed Clusters for data-intensive Applications

MapReduce Across Distributed Clusters for Data-intensive App...

引用

26th IEEE International Parallel and Distributed Processing Symposium (IPDPS) / Workshop on High Performance data intensive computing

作者： Wang, Lizhe Tao, Jie Marten, Holger Streit, Achim Khan, Samee U. Kolodziej, Joanna Chen, Dan Chinese Acad Sci Ctr Earth Observat & Digital Earth Beijing 100864 Peoples R China Steinbuch Ctr Comp Karlsruhe Inst Technol Karlsruhe Germany North Dakota State Univ Dept Elect & Comp Engn Fargo ND 58105 USA Univ Bielsko Biala Dept Math & Comp Sci Bielsko Biala Poland China Univ Geosci Sch Comp Beijing Peoples R China

ISBN: (纸本)9780769546766

Recently, the computational requirements for largescale data-intensive analysis of scientific data have grown significantly. In High Energy Physics (HEP) for example, the Large Hadron Collider (LHC) produced 13 petabytes of data in 2010. This huge amount of data are processed on more than 140 computing centers distributed across 34 countries. The MapReduce paradigm has emerged as a highly successful programming model for large-scale data-intensive computing applications. However, current MapReduce implementations are developed to operate on single cluster environments and cannot be leveraged for large-scale distributed data processing across multiple clusters. On the other hand, workflow systems are used for distributed data processing across data centers. It has been reported that the workflow paradigm has some limitations for distributed data processing, such as reliability and efficiency. In this paper, we present the design and implementation of G-Hadoop, a MapReduce framework that aims to enable large-scale distributed computing across multiple clusters. G-Hadoop uses the Gfarm file system as an underlying file system and executes MapReduce tasks across distributed clusters. Experiments of the G-Hadoop framework on distributed clusters show encouraging results.

关键词： MapReduce Hadoop data intensive computing

来源：评论

学校读者我要写书评

暂无评论

The Design of a Community Science Cloud: The Open Science data Cloud Perspective

The Design of a Community Science Cloud: The Open Science Da...

引用

25th ACM/IEEE International Conference for High Performance computing, Networking, Storage and Analysis (SC)

作者： Grossman, Robert L. Greenway, Matthew Heath, Allison P. Powell, Ray Suarez, Rafael D. Wells, Walt White, Kevin Atkinson, Malcolm Klampanos, Iraklis Alvarez, Heidi L. Harvey, Christine Mambretti, Joe J. Univ Chicago Chicago IL 60637 USA Univ Edinburgh Edinburgh EH8 9YL Midlothian Scotland Florida Int Univ Miami FL 33199 USA Richard Stockton Coll New Jersey Galloway NJ USA Northwestern Univ Evanston IL 60208 USA

ISBN: (纸本)9780769549569;9781467362184

In this paper we describe the design, and implementation of the Open Science data Cloud, or OSDC. The goal of the OSDC is to provide petabyte-scale data cloud infrastructure and related services for scientists working with large quantities of data. Currently, the OSDC consists of more than 2000 cores and 2 PB of storage distributed across four data centers connected by 10G networks. We discuss some of the lessons learned during the past three years of operation and describe the software stacks used in the OSDC. We also describe some of the research projects in biology, the earth sciences, and social sciences enabled by the OSDC.

关键词： data intensive computing cloud computing science clouds open data

来源：评论

学校读者我要写书评

暂无评论

Improving data Analysis Performance for High-Performance computing with Integrating Statistical Metadata in Scientific datasets

Improving Data Analysis Performance for High-Performance Com...

引用

High Performance computing, Networking, Storage and Analysis (SCC), SC Companion:

作者： Jialin Liu Yong Chen Computer Science Department Texas Tech University Lubbock TX USA

ISBN: (纸本)9781467362184

Scientific datasets and libraries, such as HDF5, ADIOS, and NetCDF, have been used widely in many data intensive applications. These libraries have their special file formats and I/O functions to provide efficient access to large datasets. When the data size keeps increasing, these high level I/O libraries face new challenges. Recent studies have started to utilize database techniques such as indexing and subsetting, and data reorganization to manage the increasing datasets. In this work, we present a new approach to boost the data analysis performance, namely Fast Analysis with Statistical Metadata (FASM), via data subsetting and integrating a small amount of statistics into the original datasets. The added statistical information illustrates the data shape and provides knowledge of the data distribution; therefore the original I/O libraries can utilize these statistical metadata to perform fast queries and analyses. The proposed FASM approach is currently evaluated with the PnetCDF on Lustre file systems, but can also be implemented with other scientific libraries. The FASM can potentially lead to a new dataset design and can have an impact on big data analysis.

关键词： High performance computing data intensive computing Big data Storage systems Statistical techniques FASM

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：