检索结果-内蒙古大学图书馆

Cost-intelligent application-specific data layout optimization for parallel file systems

CLUSTER computing-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2013年第2期16卷 285-298页

作者： Song, Huaiming Yin, Yanlong Chen, Yong Sun, Xian-He Dawning Informat Ind Co Ltd R&D Ctr Beijing 100084 Peoples R China IIT Dept Comp Sci Chicago IL 60616 USA Texas Tech Univ Dept Comp Sci Lubbock TX 79409 USA

Parallel file systems have been developed in recent years to ease the I/O bottleneck of high-end computing system. These advanced file systems offer several data layout strategies in order to meet the performance goals of specific I/O workloads. However, while a layout policy may perform well on some I/O workload, it may not perform as well for another. Peak I/O performance is rarely achieved due to the complex data access patterns. data access is application dependent. In this study, a cost-intelligent data access strategy based on the application-specific optimization principle is proposed. This strategy improves the I/O performance of parallel file systems. We first present examples to illustrate the difference of performance under different data layouts. By developing a cost model which estimates the completion time of data accesses in various data layouts, the layout can better match the application. Static layout optimization can be used for applications with dominant data access patterns, and dynamic layout selection with hybrid replications can be used for applications with complex I/O patterns. Theoretical analysis and experimental testing have been conducted to verify the proposed cost-intelligent layout approach. Analytical and experimental results show that the proposed cost model is effective and the application-specific data layout approach can provide up to a 74% performance improvement for data-intensive applications.

关键词： data layout I/O performance modeling Parallel file systems Parallel I/O data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

Resolvable Designs for Speeding Up Distributed computing

引用

IEEE-ACM TRANSACTIONS ON NETWORKING 2020年第4期28卷 1657-1670页

作者： Konstantinidis, Konstantinos Ramamoorthy, Aditya Iowa State Univ Dept Elect & Comp Engn Ames IA 50011 USA

Distributed computing frameworks such as MapReduce are often used to process large computational jobs. They operate by partitioning each job into smaller tasks executed on different servers. The servers also need to exchange intermediate values to complete the computation. Experimental evidence suggests that this so-called Shuffle phase can be a significant part of the overall execution time for several classes of jobs. Prior work has demonstrated a natural tradeoff between computation and communication whereby running redundant copies of jobs can reduce the Shuffle traffic load, thereby leading to reduced overall execution times. For a single job, the main drawback of this approach is that it requires the original job to be split into a number of files that grows exponentially in the system parameters. When extended to multiple jobs (with specific function types), these techniques suffer from a limitation of a similar flavor, i.e., they require an exponentially large number of jobs to be executed. In practical scenarios, these requirements can significantly reduce the promised gains of the method. In this work, we show that a class of combinatorial structures called resolvable designs can be used to develop efficient coded distributed computing schemes for both the single and multiple job scenarios considered in prior work. We present both theoretical analysis and exhaustive experimental results (on Amazon EC2 clusters) that demonstrate the performance advantages of our method. For the single and multiple job cases, we obtain speed-ups of 4.69x (and 2.6x over prior work) and 4.31x over the baseline approach, respectively.

关键词： Servers Task analysis IEEE transactions Protocols Machine learning Redundancy MapReduce data-intensive computing coded multicasting communication load TeraSort aggregate functions distributed learning

来源：评论

学校读者我要写书评

暂无评论

The HaLoop approach to large-scale iterative data analysis

引用

VLDB JOURNAL 2012年第2期21卷 169-190页

作者： Bu, Yingyi Howe, Bill Balazinska, Magdalena Ernst, Michael D. Univ Calif Irvine Irvine CA 92697 USA Univ Washington Seattle WA 98195 USA

The growing demand for large-scale data mining and data analysis applications has led both industry and academia to design new types of highly scalable data-intensive computing platforms. MapReduce has enjoyed particular success. However, MapReduce lacks built-in support for iterative programs, which arise naturally in many applications including data mining, web ranking, graph analysis, and model fitting. This paper (This is an extended version of the VLDB 2010 paper "HaLoop: Efficient Iterative data Processing on Large Clusters" PVLDB 3(1):285-296, 2010.) presents HaLoop, a modified version of the Hadoop MapReduce framework, that is designed to serve these applications. HaLoop allows iterative applications to be assembled from existing Hadoop programs without modification, and significantly improves their efficiency by providing inter-iteration caching mechanisms and a loop-aware scheduler to exploit these caches. HaLoop retains the fault-tolerance properties of MapReduce through automatic cache recovery and task re-execution. We evaluated HaLoop on a variety of real applications and real datasets. Compared with Hadoop, on average, HaLoop improved runtimes by a factor of 1.85 and shuffled only 4 % as much data between mappers and reducers in the applications that we tested.

关键词： Large-scale analytics data-intensive computing Mapreduce Cloud data management

来源：评论

学校读者我要写书评

暂无评论

Automatic optimization of stream programs via source program operator graph transformations

引用

DISTRIBUTED AND PARALLEL dataBASES 2013年第4期31卷 543-599页

作者： Dayarathna, Miyuru Suzumura, Toyotaro Tokyo Inst Technol Dept Comp Sci Meguro Ku Tokyo 1528552 Japan Tokyo Inst Technol Dept Comp Sci IBM Res Tokyo Meguro Ku Tokyo 1528552 Japan

Distributed data stream processing is a data analysis paradigm where massive amounts of data produced by various sources are analyzed online within real-time constraints. Execution performance of a stream program/query executed on such middleware is largely dependent on the ability of the programmer to fine tune the program to match the topology of the stream processing system. However, manual fine tuning of a stream program is a very difficult, error prone process that demands huge amounts of programmer time and expertise which are expensive to obtain. We describe an automated process for stream program performance optimization that uses semantic preserving automatic code transformation to improve stream processing job performance. We first identify the structure of the input program and represent the program structure in a Directed Acyclic Graph. We transform the graph using the concepts of Tri-OP Transformation and Bi-Op Transformation. The resulting sample program space is pruned using both empirical as well as profiling information to obtain a ranked list of sample programs which have higher performance compared to their parent program. We successfully implemented this methodology on a prototype stream program performance optimization mechanism called Hirundo. The mechanism has been developed for optimizing SPADE programs which run on System S stream processing run-time. Using five real world applications (called VWAP, CDR, Twitter, Apnoea, and Bargain) we show the effectiveness of our approach. Hirundo was able to identify a 31.1 times higher performance version of the CDR application within seven minutes time on a cluster of 4 nodes.

关键词： Stream processing Performance optimization Code transformation data-intensive computing Automatic tuning

来源：评论

学校读者我要写书评

暂无评论

Facilitating external sorting on SMR-based large-scale storage systems

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2021年 116卷 333-348页

作者： Chen, Chih-Hsuan Chen, Shuo-Han Liang, Yu-Pei Chen, Tseng-Yi Hsu, Tsan-sheng Wei, Hsin-Wen Shih, Wei-Kuan Natl Tsing Hua Univ Dept Comp Sci Hsinchu Taiwan Natl Taipei Univ Technol Dept Comp Sci & Informat Engn Taipei Taiwan Acad Sinica Inst Informat Sci & Informat Engn Taipei Taiwan Natl Cent Univ Dept Comp Sci & Informat Engn Taoyuan Taiwan Tamkang Univ Dept Elect & Comp Engn New Taipei Taiwan

In the big data era, retaining the capability to process and store the sheer amount of data has become a necessity for data-intensive computing. To meet the requirement of big data processing, the storage centric computing concept of processing data within storage devices has gained its popularity over the years, because the latency and energy consumed by moving data between host systems and storage devices gradually exceed that of processing data. To process data for data-intensive computing, one of the fundamental data processing technique is external sorting, which is widely used in database management systems (DBMS) and Hadoop framework. On the other hand, to store the ever-increasing volumes of data, shingled magnetic recording (SMR) drives have been proposed to increase the areal density of conventional hard disk drives (HDDs) via overlapping adjacent tracks. The SMR drive is widely regarded as a promising technology for the big data application because SMR drives can boost the capacity of HDDs without significant technology changes. Nevertheless, the overlapped track layout of SMR drive imposes the sequential write constraint on incoming write traffic, thus worsening the efficiency of performing external sorting on SMR drives. Such an observation motivates us to propose an SMR-based External Merge Sort (SMR-EMS) strategy for SMR-based large-scale storage systems with the goals of alleviating the negative impacts of sequential write constraint and enhancing the performance of external sorting on SMR drives via utilizing the concept of storage-centric computing. Experiments were conducted to demonstrate the capability of the proposed strategy on improving the efficiency of external merge sorting on SMR drives. (c) 2020 Elsevier B.V. All rights reserved.

关键词： SMR External sorting In-storage processing data-intensive computing

来源：评论

学校读者我要写书评

暂无评论

A Map-Reduce-enabled SOLAP cube for large-scale remotely sensed data aggregation

引用

COMPUTERS & GEOSCIENCES 2014年第Sep.期70卷 110-119页

作者： Li, Jiyuan Meng, Lingkui Wang, Frank Z. Zhang, Wen Cai, Yang Wuhan Univ Sch Remote Sensing & Informat Engn Wuhan 430079 Peoples R China Univ Kent Sch Comp Future Comp Grp Canterbury ME4 4AG Kent England MWR Water Resources Informat Ctr Beijing 100053 Peoples R China

Spatial On-Line Analytical Processing (SOLAP) is a powerful decision support systems tool for exploring the multidimensional perspective of spatial data. In recent years, remotely sensed data have been integrated into SOLAP cubes, and this improvement has advantages in spatio-temporal analysis for environment monitoring. However, the performance of aggregations in SOLAP still faces a considerable challenge from the large-scale dataset generated by Earth observation. From the perspective of data parallelism, a tile-based SOLAP cube model, the so-called Tile Cube, is presented in this paper. The novel model implements Roll-Up/Drill-Across operations in the SOLAP environment based on Map-Reduce, a popular data-intensive computing paradigm, and improves the throughput and scalability of raster aggregation. Therefore, the long time-series, wide-range and multi-view analysis of remotely sensed data can be processed in a short time. The Tile Cube prototype was built on Hadoop/Hbase, and drought monitoring is used as an example to illustrate the aggregations in the model. The performance testing indicated the model can be scaled along with both the data growth and node growth. It is applicable and natural to integrate the SOLAP cube with Map-Reduce. Factors that influence the performance are also discussed, and the balance of them will be considered in future works to make full use of data locality for model optimisation. (C) 2014 Elsevier Ltd. All rights reserved.

关键词： SOLAP Spatio-temporal cube data-intensive computing CyberGIS Environment monitoring

来源：评论

学校读者我要写书评

暂无评论

ASTERIX: towards a scalable, semistructured data platform for evolving-world models

引用

DISTRIBUTED AND PARALLEL dataBASES 2011年第3期29卷 185-216页

作者： Behm, Alexander Borkar, Vinayak R. Carey, Michael J. Grover, Raman Li, Chen Onose, Nicola Vernica, Rares Deutsch, Alin Papakonstantinou, Yannis Tsotras, Vassilis J. Univ Calif Irvine Irvine CA 92697 USA Univ Calif San Diego San Diego CA 92103 USA Univ Calif Riverside Riverside CA 92521 USA

ASTERIX is a new data-intensive storage and computing platform project spanning UC Irvine, UC Riverside, and UC San Diego. In this paper we provide an overview of the ASTERIX project, starting with its main goal-the storage and analysis of data pertaining to evolving-world models. We describe the requirements and associated challenges, and explain how the project is addressing them. We provide a technical overview of ASTERIX, covering its architecture, its user model for data and queries, and its approach to scalable query processing and data management. ASTERIX utilizes a new scalable runtime computational platform called Hyracks that is also discussed at an overview level;we have recently made Hyracks available in open source for use by other interested parties. We also relate our work on ASTERIX to the current state of the art and describe the research challenges that we are currently tackling as well as those that lie ahead.

关键词： data-intensive computing Cloud computing Semistructured data ASTERIX Hyracks

来源：评论

学校读者我要写书评

暂无评论

data-intensive applications, challenges, techniques and technologies: A survey on Big data

引用

INFORMATION SCIENCES 2014年 275卷 314-347页

作者： Chen, C. L. Philip Zhang, Chun-Yang Univ Macau Fac Sci & Technol Dept Comp & Informat Sci Macau Peoples R China

It is already true that Big data has drawn huge attention from researchers in information sciences, policy and decision makers in governments and enterprises. As the speed of information growth exceeds Moore's Law at the beginning of this new century, excessive data is making great troubles to human beings. However, there are so much potential and highly useful values hidden in the huge volume of data. A new scientific paradigm is born as data-intensive scientific discovery (DISD), also known as Big data problems. A large number of fields and sectors, ranging from economic and business activities to public administration, from national security to scientific researches in many areas, involve with Big data problems. On the one hand, Big data is extremely valuable to produce productivity in businesses and evolutionary breakthroughs in scientific disciplines, which give us a lot of opportunities to make great progresses in many fields. There is no doubt that the future competitions in business productivity and technologies will surely converge into the Big data explorations. On the other hand, Big data also arises with many challenges, such as difficulties in data capture, data storage, data analysis and data visualization. This paper is aimed to demonstrate a close-up view about Big data, including Big data applications, Big data opportunities and challenges, as well as the state-of-the-art techniques and technologies we currently adopt to deal with the Big data problems. We also discuss several underlying methodologies to handle the data deluge, for example, granular computing, cloud computing, bio-inspired computing, and quantum computing. (C) 2014 Elsevier Inc. All rights reserved.

关键词： Big data data-intensive computing e-Science Parallel and distributed computing Cloud computing

来源：评论

学校读者我要写书评

暂无评论

Toward a Microarchitecture for Efficient Execution of Irregular Applications

引用

ACM TRANSACTIONS ON PARALLEL computing 2020年第4期7卷 1–24页

作者： Leidel, John D. Wang, Xi Williams, Brody Chen, Yong Tact Comp Labs Box 43104 Lubbock TX 79409 USA Texas Tech Univ Box 43104 Lubbock TX 79409 USA

Given the increasing importance of efficient data-intensive computing, we find that modern processor designs are not well suited to the irregular memory access patterns often found in these algorithms. Applications and algorithms that do not exhibit spatial and temporal memory request locality induce high latency and low memory bandwidth due to the high cache miss rate. In response to the performance penalties inherently present in applications with irregular memory accesses, we introduce a GoblinCore-64 (GC64) architecture and a unique memory hierarchy that are explicitly designed to exploit memory performance from irregular memory access patterns. GC64 provides a pressure-driven hardware-managed concurrency control to minimize pipeline stalls and lower the latency of context switches. A novel memory coalescing model is also introduced to enhance the performance of memory systems via request aggregations. We have evaluated the performance benefits of our approach using a series of 24 benchmarks and the results show nearly 50% memory request reductions and a performance acceleration of up to 14.6x.

关键词： data-intensive computing irregular algorithms thread concurrency context switching

来源：评论

学校读者我要写书评

暂无评论

An improved task assignment scheme for Hadoop running in the clouds

引用

JOURNAL OF CLOUD computing-ADVANCES SYSTEMS AND APPLICATIONS 2013年第1期2卷 1-16页

作者： Dai, Wei Bassiouni, Mostafa Univ Cent Florida Sch Elect Engn & Comp Sci 4000 Cent Florida Blvd Orlando FL 32816 USA

Nowadays, data-intensive problems are so prevalent that numerous organizations in various industries have to face them in their business operation. It is often crucial for enterprises to have the capability of analyzing large volumes of data in an effective and timely manner. MapReduce and its open-source implementation Hadoop dramatically simplified the development of parallel data-intensive computing applications for ordinary users, and the combination of Hadoop and cloud computing made large-scale parallel data-intensive computing much more accessible to all potential users than ever before. Although Hadoop has become the most popular data management framework for parallel data-intensive computing in the clouds, the Hadoop scheduler is not a perfect match for the cloud environments. In this paper, we discuss the issues with the Hadoop task assignment scheme, and present an improved scheme for heterogeneous computing environments, such as the public clouds. The proposed scheme is based on an optimal minimum makespan algorithm. It projects and compares the completion times of all task slots' next data block, and explicitly strives to shorten the completion time of the map phase of MapReduce jobs. We conducted extensive simulation to evaluate the performance of the proposed scheme compared with the Hadoop scheme in two types of heterogeneous computing environments that are typical on the public cloud platforms. The simulation results showed that the proposed scheme could remarkably reduce the map phase completion time, and it could reduce the amount of remote processing employed to a more significant extent which makes the data processing less vulnerable to both network congestion and disk contention.

关键词： Cloud computing Hadoop MapReduce Task assignment data-intensive computing Parallel and distributed computing

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：