检索结果-内蒙古大学图书馆

data prefetching and file synchronizing for performance optimization in Hadoop-based hybrid cloud

JOURNAL OF SYSTEMS AND SOFTWARE 2019年 151卷 133-149页

作者： Li, Chunlin Zhang, Jing Chen, Yi Luo, Youlong Wuhan Univ Technol Sch Comp Sci & Technol Wuhan 430063 Hubei Peoples R China Huanghuai Univ Int Coll Zhumadian 463000 Peoples R China Beijing Technol & Business Univ Beijing Key Lab Big Data Technol Food Safety Beijing Peoples R China Xian Univ Posts & Telecommun Shaanxi Key Lab Network Data Anal & Intelligent P Xian 710121 Shaanxi Peoples R China

Driven by the technical factors such as system reliability, bandwidth constraints, data confidentiality and security, as well as the economic factors such as initial capital expenditure and re-occurring operating expenditure, today's cloud computing tends to adopt hybrid cloud model. However, because hybrid clouds scale both numerically and geographically, the network delay becomes the main constraint in remote file system access. To hide network latency and reduce job completion time in Hadoop-based hybrid cloud data access, a scheduling-aware data prefetching scheme to enhance non-local map task's data locality in Hadoop-based centralized hybrid cloud (CHCDLOS-Prefetch) and a file synchronizing method to decrease job execution delay in Hadoop-based distributed hybrid cloud (DHCDLO-Sync) are proposed. In the former, input data for non-local map tasks are fetched ahead of time to target compute nodes by making use of idle network bandwidth. In the latter, considered from job level scheduling, data files with high popularity are proactively synchronized beforehand among sub-clouds to strength intra sub-cloud data locality in distributed hybrid cloud. Extensive experimental results illustrate that compared to the Capacity, the Fair and the DARE algorithms, our proposed algorithms improve hybrid cloud performance more significantly in data locality and job completion time. (C) 2019 Elsevier Inc. All rights reserved.

关键词： data prefetching File synchronizing Hybrid cloud

来源：评论

学校读者我要写书评

暂无评论

data prefetching and Eviction Mechanisms of In-Memory Storage Systems Based on Scheduling for Big data Processing

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2019年第8期30卷 1738-1752页

作者： Chen, Chien-Hung Hsia, Ting-Yuan Huang, Yennun Kuo, Sy-Yen Natl Taiwan Univ Dept Elect Engn Taipei 10617 Taiwan Acad Sinica Res Ctr Informat Technol Innovat Taipei 11529 Taiwan

In-memory techniques keep data into faster and more expensive storage media for improving performance of big data processing. However, existing mechanisms do not consider how to expedite the data processing applications that access the input datasets only once. Another problem is how to reclaim memory without affecting other running applications. In this paper, we provide scheduling-aware data prefetching and eviction mechanisms based on Spark, Alluxio, and Hadoop. The mechanisms prefetch data and release memory resources based on the scheduling information. A mathematical method is proposed for maximizing the reduction of data access time. To make the mechanisms applicable in large-scale environments, we propose a heuristic algorithm to reduce the computational time. Furthermore, an enhanced version of the heuristic algorithm is also proposed to increase the amount of prefetched data. Finally, we perform real-testbed and simulation experiments to show the effectiveness of the proposed mechanisms.

关键词： Big data processing in-memory systems scheduling information data prefetching data eviction

来源：评论

学校读者我要写书评

暂无评论

Taxonomy of data prefetching for Multicore Processors

引用

Journal of Computer Science & Technology 2009年第3期24卷 405-417页

作者： Surendra Byna 陈勇孙贤和 Department of Computer Science Illinois Institute of TechnologyChicagoIllinois 60616U.S.A.

data prefetching is an effective data access latency hiding technique to mask the CPU stall caused by cache misses and to bridge the performance gap between processor and memory. With hardware and/or software support, data prefetching brings data closer to a processor before it is actually needed. Many prefetching techniques have been developed for single-core processors. Recent developments in processor technology have brought multicore processors into mainstream. While some of the single-core prefetching techniques are directly applicable to multicore processors, numerous novel strategies have been proposed in the past few years to take advantage of multiple cores. This paper aims to provide a comprehensive review of the state-of-the-art prefetching techniques, and proposes a taxonomy that classifies various design concerns in developing a prefetching strategy, especially for multicore processors. We compare various existing methods through analysis as well.

关键词： taxonomy of prefetching strategies multicore processors data prefetching memory hierarchy

来源：评论

学校读者我要写书评

暂无评论

data prefetching for Heterogeneous Hadoop Cluster 5

Data Prefetching for Heterogeneous Hadoop Cluster

引用

5th International Conference on Advanced Computing and Communication Systems (ICACCS)

作者： Vinutha, D. C. Raju, G. T. RNS Inst Technol Bengaluru Dept CSE Bengaluru India Vidyavardhaka Coll Engn Dept ISE Mysuru India Visvesraya Technol Univ Belagavi Karnataka India

ISBN: (纸本)9781538695333

: Hadoop is an open source implementation of MapReduce. Performance of Hadoop is affected by the overhead of communication during the transmission of large datasets to the computing node. In a heterogeneous cluster if a map task wants to process the data, which is not present in the local disk then the data transmission overhead occurs. To overcome this issue, data prefetching for Heterogeneous Hadoop cluster for resource optimization is proposed. data prefetching is used to fetch the input data in advance from the remote node to a particular computing node. Hence transmission of data occurs in parallel with data processing and the job execution time is reduced. Different MapReduce jobs are used to conduct the experiment. The results demonstrate that the time taken to transmit the data is reduced. The job execution time is reduced by 15% for the input data size greater than or equal to 2GB and performance improvement of 25% is obtained for 64MB block size.

关键词： Hadoop data transmission MapReduce data prefetching data Processing

来源：评论

学校读者我要写书评

暂无评论

data prefetching on in-order processors 16

Data prefetching on in-order processors

引用

International Conference on High Performance Computing & Simulation (HPCS)

作者： Ortega, Cristobal Garcia, Victor Moreto, Miquel Casas, Marc Rusitoru, Roxana Univ Politecn Cataluna Barcelona Spain Barcelona Supercomp Ctr BSC CNS Barcelona Spain ARM Ltd Cambridge England

ISBN: (纸本)9781538678794

Low-power processors have attracted attention due to their energy-efficiency. A large market, such as the mobile one, relies on these processors for this very reason. Even High Performance Computing (HPC) systems are starting to consider low-power processors as a way to achieve exascale performance within 20MW, however, they must meet the right performance/Watt balance. Current low-power processors contain in-order cores, which cannot re-order instructions to avoid data dependency-induced stalls. Whilst this is useful to reduce the chip's total power consumption, it brings several challenges. Due to the evolving performance gap between memory and processor, memory is a significant bottleneck. In-order cores cannot reorder instructions and are memory latency bound, something data prefetching can help alleviate by ensuring data is readily available. In this work, we do an exhaustive analysis of available data prefetching techniques in state-of-the-art in-order cores. We analyze 5 static prefetchers and 2 dynamic aggressiveness and destination mechanisms applied to 3 data prefetchers on a set of HPC mini- and proxy-applications, whilst running on in-order processors. We show that next-line prefetching can achieve nearly top performance with a reasonable bandwidth consumption when throttled, whilst neighbor prefetchers have been found to be best, overall.

关键词： data prefetching in-order processor High Perfomance Computing

来源：评论

学校读者我要写书评

暂无评论

Cost-based data prefetching and Scheduling in Big data Platforms over Tiered Storage Systems

引用

ACM TRANSACTIONS ON dataBASE SYSTEMS 2023年第4期48卷 1-40页

作者： Herodotou, Herodotos Kakoulli, Elena Cyprus Univ Technol 30 Arch Kyprianos Str CY-3036 Limassol Cyprus Neapolis Univ Pafos 2 Danais Ave CY-8042 Pafos Cyprus

The use of storage tiering is becoming popular in data-intensive compute clusters due to the recent advancements in storage technologies. The Hadoop Distributed File System, for example, now supports storing data in memory, SSDs, and HDDs, while OctopusFS and hatS offer fine-grained storage tiering solutions. However, current big data platforms (such as Hadoop and Spark) are not exploiting the presence of storage tiers and the opportunities they present for performance optimizations. Specifically, schedulers and prefetchers will make decisions only based on data locality information and completely ignore the fact that local data are now stored on a variety of storage media with different performance characteristics. This article presents Trident, a scheduling and prefetching framework that is designed to make task assignment, resource scheduling, and prefetching decisions based on both locality and storage tier information. Trident formulates task scheduling as aminimum cost maximummatching problem in a bipartite graph and utilizes two novel pruning algorithms for bounding the size of the graph, while still guaranteeing optimality. In addition, Trident extends YARN's resource request model and proposes a new storage-tier-aware resource scheduling algorithm. Finally, Trident includes a cost-based data prefetching approach that coordinates with the schedulers for optimizing prefetching operations. Trident is implemented in both Spark and Hadoop and evaluated extensively using a realistic workload derived from Facebook traces as well as an industry-validated benchmark, demonstrating significant benefits in terms of application performance and cluster efficiency.

关键词： Distributed file systems tiered storage data prefetching task scheduling

来源：评论

学校读者我要写书评

暂无评论

Deep learning based data prefetching in CPU-GPU unified virtual memory

引用

JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING 2023年第1期174卷 19-31页

作者： Long, Xinjian Gong, Xiangyang Zhang, Bo Zhou, Huiyang Beijing Univ Posts & Telecommun State Key Lab Networking & Switching Technol Beijing 100876 Peoples R China Beijing Univ Posts & Telecommun Sch Comp Sci Natl Pilot Software Engn Sch Beijing 100876 Peoples R China North Carolina State Univ Dept Elect & Comp Engn Raleigh NC 27606 USA

Unified Virtual Memory (UVM) relieves the developers from the onus of maintaining complex data structures and explicit data migration by enabling on-demand data movement between CPU memory and GPU memory. However, on-demand paging soon becomes a performance bottleneck of UVM due to the high latency caused by page table walks and data migration over interconnect. prefetching is considered a promising solution to this problem given its ability to leverage the locality of program memory access patterns. However, existing locality-based prefetching schemes can not handle all the situations. An ideal prefetcher should not only look at narrow regions of the requested address space but also capture global context to deliver a good prediction of the memory access pattern. This paper proposes a novel framework for page prefetching for UVM through deep learning. We first show that a powerful Transformer learning model can provide high accuracy for UVM page prefetching. We then perform analysis to interpret this Transformer model and derive several insights that allow us to design a simpler model to match the unconstrained model's accuracy with orders of magnitude lower cost. We use a pattern-based method to make the UVM page preditor general to different GPU workloads. We evaluate this framework on a set of 11 memory-intensive benchmarks from popular benchmark suites. Our solution outperforms the state-of-the-art (SOTA) UVM framework, improving the performance by 10.89%, improving the device memory page hit rate by 16.98% (89.02% vs. 76.10% for prior art), and reducing the CPU-GPU interconnect traffic by 11.05%. According to our proposed unified metric, which combines the accuracy, coverage, and page hit rate, our solution is approaching the ideal prefetching scheme more than the SOTA design (0.90 vs. 0.85, with the perfect prefetcher of 1.0).(c) 2022 Elsevier Inc. All rights reserved.

关键词： data prefetching Graphics processing unit Unified virtual memory Deep learning Transformer

来源：评论

学校读者我要写书评

暂无评论

WASP: Selective data prefetching with Monitoring Runtime Warp Progress on GPUs

引用

IEEE TRANSACTIONS ON COMPUTERS 2018年第9期67卷 1366-1373页

作者： Oh, Yunho Yoon, Myung Kuk Park, Jong Hyun Park, Yongjun Ro, Won Woo Yonsei Univ Sch Elect & Elect Engn Seoul South Korea Hanyang Univ Div Comp Sci & Engn Seoul South Korea

This paper proposes a new data prefetching technique for Graphics Processing Units (GPUs) called Warp Aware Selective prefetching (WASP). The main idea of WASP is to dynamically select warps whose progress is slower than that of the current warp as prefetching target warps. Under the in-order instruction execution model of GPUs, these prefetching target warps will certainly execute the same load as the current warp. Exploiting that, WASP prefetches the data for prefetching target warps, which allows the prefetched data to be accurately accessed. To simply verify the progress of the warps, WASP monitors the counts of the dynamic load executions for all warps. When a warp executes a load, WASP searches the warps with lower load execution counts than the current warp and generates the prefetch requests for them. In our evaluation, WASP achieves a 16.8 percent speedup compared to the baseline GPU.

关键词： GPGPU data prefetching warp scheduling cache performance

来源：评论

学校读者我要写书评

暂无评论

Image parallel ray tracing using static load balancing and data prefetching

引用

PARALLEL COMPUTING 1997年第7期23卷 861-872页

作者： Yoon, HJ Eun, S Cho, JW HAN NAM UNIV DEPT INFORMAT COMMUN ENGNDAEDUK KUTAEJON 300791SOUTH KOREA

We propose an image parallel ray tracing with a static load balancing and data prefetching scheme on distributed memory multicomputers. We use a scattered decomposition of which the allocation unit is one pixel and each processor lakes part in pixels scattered around the whole image space, which can make a load balance efficiently. In a scattered decomposition scheme, remote data accesses may be frequent since it can not utilize ray coherence sufficiently. To reduce the communication overhead due to this, we propose data prefetching by multicasting (DPM) where requested object data are also transferred to the processors dealing with the adjacent pixels if a processor requests a remote data. We implemented our approach on the Intel's Paragon and showed the linear speedup. However, the advantages of data prefetching did not materialize due to the machine's inefficient multicast operation.

关键词： ray tracing static load balancing data prefetching parallel rendering

来源：评论

学校读者我要写书评

暂无评论

An efficient Markov chain-based data prefetching for motion estimation of HEVC on multi-core DSPs

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2016年第13期75卷 8019-8043页

作者： Sun, Shiming Jiang, Hongxu Li, Bo Beihang Univ Beijing Key Lab Digital Media Sch Comp Sci & Engn Beijing 100191 Peoples R China China Univ Petr Sch Comp Sci & Commun Engn Qingdao 266580 Peoples R China

As one of the most time-consuming parts of video coding, Motion Estimation (ME) has always been the major issue in the embedded coding system due to its memory-intensive nature. This is even truer now as the gap between processor and memory speed continues to grow in the embedded coding system on multi-core processors. In this paper, a data prefetching algorithm based on a Markov Chain Model (MCMDP) is presented to improve the data access efficiency for the ME of High Efficiency Video Coding (HEVC) on multi-core DSPs. First, by analyzing the process and features of ME, a new method of calculating Motion Vector Predictions (MVPs) is given, in which the coding block's MVP is estimated from the MVPs of the reference picture instead of the motion vectors of the neighboring blocks. This is critical to improve the efficiency of data prefetching for ME because it eliminates the data dependencies that cause the latency of data prefetching. Second, the experimental results show that the probability distribution of the search windows in ME has continuity and locality in successive pictures, and these statistical properties are consistent with the characteristics of Markov chains. Therefore, a new model based on Markov chains is designed for predicting the prefetch window that covers the search window to improve the coverage of data prefetching. Finally, the experiments on TMS320C6678 demonstrate that the prefetching efficiency is significantly improved for the ME of HEVC on multi-core DSPs.

关键词： Motion estimation data prefetching Markov chain-based model HEVC Multi-core DSP

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：