检索结果-内蒙古大学图书馆

Segmented In-Advance data Analytics for Fast Scientific Discovery

IEEE TRANSACTIONS ON CLOUD computing 2020年第2期8卷 432-442页

作者： Liu, Jialin Chen, Yong Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

Scientific discovery usually involves data generation, data preprocessing, data storage and data analysis. As the data volume exceeds a few terabytes (TB) in a single simulation run, the data movement, which happens during each cycle of the scientific discovery, continues to be the bottleneck in most scientific big data applications. A lot of research works have been conducted on reducing the data movement. Among the existing efforts and based on our previous research, reusing the analysis results shows a significant potential in optimizing the data movement between analysis operations. In this work, we propose the Segmented In-Advance (SIA) data analytics approach for optimizing the data movement and we also provide a cloud-based elastic distributed in-memory database to manage the intermediate analysis results. The fundamental idea of this Segmented In-Advance approach is to analyze the history operations and to predict the future interesting analytics operations. The predicted analysis operation is in-advance performed on the finer segmented dataset and the segmented results are distributed in an in-memory key-value store for future reuse. The evaluation shows that the segmented in-advance data analytics approach achieves 1.2X-6.1X speedup. The evaluation also shows a good scalability of the in-memory distributed data store. The proposed Segmented In-Advance data analytics approach is a promising data movement reduction solution for scientific big data applications and fast scientific discovery.

关键词： Segmented in-advance data analytics big data data intensive computing scientific computing

来源：评论

学校读者我要写书评

暂无评论

Analyzing design choices for distributed multidimensional indexing

引用

JOURNAL OF SUPERcomputing 2012年第3期59卷 1552-1576页

作者： Nam, Beomseok Sussman, Alan Ulsan Natl Inst Sci & Technol Sch Elect & Comp Engn Ulsan 689798 South Korea Univ Maryland Dept Comp Sci College Pk MD 20742 USA

Scientific datasets are often stored on distributed archival storage systems, because geographically distributed sensor devices store the datasets in their local machines and also because the size of scientific datasets demands large amount of disk space. Multidimensional indexing techniques have been shown to greatly improve range query performance into large scientific datasets. In this paper, we discuss several ways of distributing a multidimensional index in order to speed up access to large distributed scientific datasets. This paper compares the designs, challenges, and problems for distributed multidimensional indexing schemes, and provides a comprehensive performance study of distributed indexing to provide guidelines to choose a distributed multidimensional index for a specific data analysis application.

关键词： Multidimensional indexing Distributed indexing Decentralized indexing data intensive computing

来源：评论

学校读者我要写书评

暂无评论

TRANSFORMER: A NEW PARADIGM FOR BUILDING data-PARALLEL PROGRAMMING MODELS

引用

IEEE MICRO 2010年第4期30卷 55-64页

作者： Wang, Peng Meng, Dan Han, Jizhong Zhan, Jianfeng Tu, Bibo Shi, Xiaofeng Wan, Le Chinese Acad Sci Inst Comp Technol Beijing 100190 Peoples R China

Cloud computing drives the design and development of diverse programming models for massive data processing. the transformer programming framework aims to facilitate the building of diverse data-parallel programming models. Transformer has two layers: a common runtime system and a model-specific system. using transformer, the authors show how to implement three programming models: dryad-like data flow, MapReduce, and all-pairs.

关键词： actor model All-Pairs cloud computing Computational modeling data flow data intensive computing data models Libraries Load modeling MapReduce Programming programming model Receivers Runtime

来源：评论

学校读者我要写书评

暂无评论

ParSA: High-throughput scientific data analysis framework with distributed file system

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2015年 51卷 111-119页

作者： Zhang, Tao Sun, XiangZheng Xue, Wei Qiao, Nan Huang, Huang Shu, JiWu Zheng, Weimin Tsinghua Univ Dept Comp Sci & Technol Beijing 100084 Peoples R China Intel Corp Beijing Peoples R China

Scientific data analysis and visualization have become the key component for nowadays large scale simulations. Due to the rapidly increasing data volume and awkward I/O pattern among high structured files, known serial methods/tools cannot scale well and usually lead to poor performance over traditional architectures. In this paper, we propose a new framework: ParSA (parallel scientific data analysis) for high-throughput and scalable scientific analysis, with distributed file system. ParSA presents the optimization strategies for grouping and splitting logical units to utilize distributed I/O property of distributed file system, scheduling the distribution of block replicas to reduce network reading, as well as to maximize overlapping the data reading, processing, and transferring during computation. Besides, ParSA provides the similar interfaces as the NetCDF Operator (NCO), which is used in most of climate data diagnostic packages, making it easy to use this framework. We utilize ParSA to accelerate well-known analysis methods for climate models on Hadoop Distributed File System (HDFS). Experimental results demonstrate the high efficiency and scalability of ParSA, getting the maximum 1.3 GB/s throughput on a six nodes Hadoop cluster with five disks per node. Yet, it can only get 392 MB/s throughput on a RAID-6 storage node. (C) 2014 Elsevier B.V. All rights reserved.

关键词： data intensive computing Scientific data analysis Distributed file system

来源：评论

学校读者我要写书评

暂无评论

Hierarchical Collective I/O Scheduling for High-Performance computing

引用

BIG data RESEARCH 2015年第3期2卷 117-126页

作者： Liu, Jialin Zhuang, Yu Chen, Yong Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

The non-contiguous access pattern of many scientific applications results in a large number of I/O requests, which can seriously limit the data-access performance. Collective I/O has been widely used to address this issue. However, the performance of collective I/O could be dramatically degraded in today's high-performance computing systems due to the increasing shuffle cost caused by highly concurrent data accesses. This situation tends to be even worse as many applications become more and more data intensive. Previous research has primarily focused on optimizing I/O access cost in collective I/O but largely ignored the shuffle cost involved. Previous works assume that the lowest average response time leads to the best QoS and performance, while that is not always true for collective requests when considering the additional shuffle cost. In this study, we propose a new hierarchical I/O scheduling (HIO) algorithm to address the increasing shuffle cost in collective I/O. The fundamental idea is to schedule applications' I/O requests based on a shuffle cost analysis to achieve the optimal overall performance, instead of achieving optimal I/O accesses only. The algorithm is currently evaluated with the MPICH3 and PVFS2. Both theoretical analysis and experimental tests show that the proposed hierarchical I/O scheduling has a potential in addressing the degraded performance issue of collective I/O with highly concurrent accesses. (C) 2015 Elsevier Inc. All rights reserved.

关键词： Collective I/O Scheduling High-performance computing Big data data intensive computing

来源：评论

学校读者我要写书评

暂无评论

Implementing data cube construction using a cluster middleware: algorithms, implementation experience, and performance evaluation

引用

FUTURE GENERATION COMPUTER SYSTEMS 2003年第4期19卷 533-550页

作者： Yang, G Jin, RM Agrawal, G Ohio State Univ Dept Comp & Informat Sci Columbus OH 43210 USA

With increases in the amount of data available for analysis in commercial settings, on line analytical processing (OLAP) and decision support have become important applications for high performance computing. Implementing such applications on clusters requires a lot of expertise and effort, particularly because of the sizes of input and output datasets. In this paper, we describe our experiences in developing one such application using a cluster middleware, called ADR. We focus on the problem of data cube construction, which commonly arises in multi-dimensional OLAP. We show how ADR, originally developed for scientific data intensive applications, can be used for carrying out an efficient and scalable data cube construction implementation. A particular issue with the use of ADR is tiling of output datasets. We present new algorithms that combine interprocessor communication and tiling within each processor. These algorithms preserve the important properties that are desirable from any parallel data cube construction algorithm. We have carried out a detailed evaluation of our implementation. The main results from our experiments are as follows: (1) high speedups are achieved on both dense and sparse datasets, even though we have used simple algorithms that sequentialize a part of the computation;(2) the execution time depends only upon the amount of computation, and does not increase in a super-linear fashion as the dataset size or the number of tiles increases;and (3) as the datasets become more sparse, sequential performance degrades, but the parallel speedups are still quite good. As part of our on-going work in this area, we are also looking at handling a larger number of dimensions and multi-dimensional partitionings. We describe our preliminary theoretical and experimental work in this direction. (C) 2003 Elsevier Science B.V. All rights reserved.

关键词： data cube construction cluster middleware data intensive computing performance evaluation

来源：评论

学校读者我要写书评

暂无评论

Management of grid jobs and data within SAMGrid

Management of grid jobs and data within SAMGrid

引用

IEEE International Conference on Cluster computing

作者： Baranovski, A Garzoglio, G Terekhov, I Roy, A Tannenbaum, T Fermilab Natl Accelerator Lab Batavia IL 60510 USA

ISBN: (纸本)0780386949

When designing SAMGrid, a project for distributing high-energy physics computations on. a grid, we discovered that it was challenging to decide where to place user's jobs. Jobs typically need to access hundreds of files, and each site has a different subset of the files. Our data system SAM knows what portion of a user's data may be at each site, but does not know how to submit grid jobs. Our job submission system Condor-G knows how to submit grid jobs, but originally it required users to choose grid sites and gave them no assistance in choosing. This paper describes how we enhanced Condor G to interact with SAM to make good decisions about where jobs should be executed, and thereby improve the performance of grid jobs that access large amounts of data. All these enhancements are general enough to be applicable to grid computing beyond the data-intensive computing with SAMGrid.

关键词： SAMGrid Condor-G SAM DO grid computing planning middleware data management scheduling data intensive computing

来源：评论

学校读者我要写书评

暂无评论

Statistics-based Workload Modeling for MapReduce

Statistics-based Workload Modeling for MapReduce

引用

26th IEEE International Parallel and Distributed Processing Symposium (IPDPS) / Workshop on High Performance data intensive computing

作者： Yang, Hailong Luan, Zhongzhi Li, Wenjun Qian, Depei Guan, Gang Beihang Univ Sch Comp Sci Beijing Peoples R China Tencent Res Beijing Peoples R China

ISBN: (纸本)9780769546766

Large-scale data-intensive computing with MapReduce framework in Cloud is becoming pervasive for the core business of many academic, government, and industrial organizations. Hadoop is by far the most successful realization of MapReduce framework. While MapReduce is easy-to-use, efficient and reliable for data-intensive computations, the excessive configuration parameters in Hadoop cause unexpected challenges when running various workloads with Hadoop cluster effectively. Consequently, developers who have less experience with the Hadoop configuration system may devote a significant effort to write an application with poor performance, because they have no idea how these configurations would influence the performance, or they are not even aware that these configurations exist. In this paper, we propose a statistic analysis approach to identify the relationships among workload characteristics, Hadoop configurations and workload performance. Several non-intuitive relationships between workload characteristics and relative performance are revealed and the experimental results demonstrate that our regression models accurately predict the performance of MapReduce workloads under different Hadoop configurations.

关键词： data intensive computing MapReduce analytical model performance prediction

来源：评论

学校读者我要写书评

暂无评论

Cross-Institutional Research Cyberinfrastructure for data intensive Science

Cross-Institutional Research Cyberinfrastructure for Data In...

引用

IEEE High Performance Extreme computing Conference (HPEC)

作者： Lenhardt, W. Christopher Conway, Mike Scott, Erik Blanton, Brian Krishnamurthy, Ashok Hadzikadic, Mirsad Vouk, Mladen Wilson, Alyson Univ N Carolina Renaissance Comp Inst RENCI Chapel Hill NC 27514 USA Univ N Carolina Charlotte NC 28223 USA North Carolina State Univ Raleigh NC 27695 USA

ISBN: (纸本)9781509035250

This paper describes a multi-institution effort to develop a "data science as a service" platform. This platform integrates advanced federated data management for small to large datasets, access to high performance computing, distributed computing and advanced networking. The goal is to develop a platform that is flexible and extensible while still supporting domain research and avoiding the walled garden problem. Some preliminary lessons learned and next steps will also be outlined.

关键词： distributed computing risk analytics distributed data open system architectures data intensive computing big data and distributed computing

来源：评论

学校读者我要写书评

暂无评论

Unified and Efficient HEC Storage System with a Working-Set based Reorganization Scheme

Unified and Efficient HEC Storage System with a Working-Set ...

引用

15th IEEE International Conference on Cluster computing (CLUSTER)

作者： Chen, Junjie Chen, Yong Texas Tech Univ Lubbock TX 79409 USA Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

ISBN: (纸本)9781479908981

High-end computing (HEC) applications and simulations have become increasingly data intensive. The pressure on the storage system capability has substantially increased in recent years. Traditional hard disk drives (HDD) are dominant storage devices in HEC, but suffer seek time delays and rotational latencies. The emerged storage class memory such as Solid State Drive (SSD) provides a new promising high bandwidth and low latency storage solution, but with inherent limitations of small capacity, limited writing cycles, and high cost. SSDs and HDDs have complement characteristics in nature and there is a desire to combine and unify them to best serve HEC workloads. In this paper, we introduce our initial study of a novel Working-Set based Reorganization Scheme (WS-ROS), to manage and leverage the merits of both SSDs and HDDs and to provide a highly efficient storage system. The goal of the proposed scheme is to leverage SSDs service more read requests through keeping active data and to leverage HDDs to keep large non-active data and extend SSDs writing cycles. The preliminary evaluations have shown that the proposed scheme has significant advantages and holds a promise as a unified storage management scheme for HEC workloads.

关键词： High-end computing data intensive computing storage parallel file systems Solid State Drives

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：