检索结果-内蒙古大学图书馆

A multi-factor monitoring fault tolerance model based on a GPU cluster for big data processing

INFORMATION SCIENCES 2019年 496卷 300-316页

作者： Fang, Yuling Chen, Qingkui Xiong, Naixue Univ Shanghai Sci & Technol Shanghai 200093 Peoples R China Northeastern State Univ Dept Math & Comp Sci Tahlequah OK 74464 USA

High-performance computing clusters are widely used in large-scale data mining applications, and have higher requirements for persistence, stability and real-time use and sre therefore computationally intensive. To support large-scale data processing, we design a multi-factor real-time monitoring fault tolerance (MRMFT) model based on a GPU cluster. However, the higher clock frequency of GPU chips results in excessively high energy consumption in computing systems. Moreover, the ability to support a long-lasting high temperature operation varies greatly between different GPUs owing to the individual differences between the chips. In this paper, we design a GPU cluster energy consumption monitoring system based on wireless sensor networks (WSNs) and propose an energy consumption aware checkpointing (ECAC) for high energy consumption problems with the following two advantages: the system sets checkpoints according to actual energy consumption and the device temperature to improve the utilization of checkpoints and reduce time cost;and it exploits the parallel computing features of CPU and GPU to hide the CPU detection overhead in GPU parallel computation, and further reduce the time and energy consumption overhead in the fault tolerance phase. Using ECAC as the constraint and aiming for a persistent and reliable operation, the dynamic task migration mechanism is designed, and the reliability of the cluster is greatly improved. The theoretical analysis and experiment results show that the model improves the persistence and stability of the computing system while reducing checkpoint overhead. (C) 2018 Elsevier Inc. All rights reserved.

关键词： big data processing GPU cluster Persistence computing Energy consumption Fault tolerance Energy consumption aware checkpointing Task migration

来源：评论

学校读者我要写书评

暂无评论

FvRS: Efficiently identifying performance-critical data for improving performance of big data processing

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2019年 91卷 157-166页

作者： Xu, Gaoxiang Tan, Zhipeng Feng, Dan Yang, Laurence T. Zhou, Wei Zhang, Xinyan Zhang, Yang Xu, Jie Huazhong Univ Sci & Technol Wuhan Natl Lab Optoelect Key Lab Informat Storage Syst Sch Comp Sci & TechnolMinist Educ China Wuhan Hubei Peoples R China St Francis Xavier Univ Dept Comp Sci Antigonish NS Canada

Hybrid storage is widely implemented in big data processing by providing large storage capacity and high access speed in an economical manner. Performance-critical data are usually stored in SSD to obtain the most performance benefits with the least storage cost. Conventional scheme identifies performance critical data based on data's access hotness. But it does not consider data's I/O cost and may store low-cost data in SSD, resulting in the waste of SSD. The recently proposed scheme determines performance-critical data based on both access hotness and I/O cost. However, it fails to accurately evaluate I/O cost, thus still distributes many low-cost data on SSD. In this paper, we propose a sequentiality-aware identification scheme for performance-critical data, called FvRS, which boots the accuracy of I/O cost evaluation by exploiting data's access sequentiality. The key idea is to evaluate data's I/O cost based on both request size and access sequentiality. By properly identifying high-cost hot data, FvRS maximizes the utilization of SSD to improve system performance. In addition, FvRS maintains performance-critical data in a real-time table to reduce the identification overhead. We have implemented FvRS in a hybrid storage system in Linux. Extensive evaluations using three real-workload traces and a famous benchmark Postmark demonstrate the accuracy and efficiency of FvRS. Compared with the state-of-the-art schemes, such as hotness-based identification and cost-based identification, FvRS reduces I/O response time by 10.3%similar to 45.6% and 16.3%similar to 25.1%, respectively. (C) 2018 Elsevier B.V. All rights reserved.

关键词： big data processing Performance-critical data I/O cost evaluation Access sequentiality Hybrid storage

来源：评论

学校读者我要写书评

暂无评论

data Prefetching and Eviction Mechanisms of In-Memory Storage Systems Based on Scheduling for big data processing

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2019年第8期30卷 1738-1752页

作者： Chen, Chien-Hung Hsia, Ting-Yuan Huang, Yennun Kuo, Sy-Yen Natl Taiwan Univ Dept Elect Engn Taipei 10617 Taiwan Acad Sinica Res Ctr Informat Technol Innovat Taipei 11529 Taiwan

In-memory techniques keep data into faster and more expensive storage media for improving performance of big data processing. However, existing mechanisms do not consider how to expedite the data processing applications that access the input datasets only once. Another problem is how to reclaim memory without affecting other running applications. In this paper, we provide scheduling-aware data prefetching and eviction mechanisms based on Spark, Alluxio, and Hadoop. The mechanisms prefetch data and release memory resources based on the scheduling information. A mathematical method is proposed for maximizing the reduction of data access time. To make the mechanisms applicable in large-scale environments, we propose a heuristic algorithm to reduce the computational time. Furthermore, an enhanced version of the heuristic algorithm is also proposed to increase the amount of prefetched data. Finally, we perform real-testbed and simulation experiments to show the effectiveness of the proposed mechanisms.

关键词： big data processing in-memory systems scheduling information data prefetching data eviction

来源：评论

学校读者我要写书评

暂无评论

mrMoulder: A recommendation-based adaptive parameter tuning approach for big data processing platform

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2019年 93卷 570-582页

作者： Cai, Lin Qi, Yong Wei, Wei Wu, Jinsong Li, Jingwei Xi An Jiao Tong Univ Sch Elect & Informat Engn Xian 710049 Shaanxi Peoples R China Xian Univ Technol Sch Comp Sci & Engn Xian 710048 Shaanxi Peoples R China Univ Chile Dept Elect Engn Av Tupper 2007 Santiago 8370451 Chile

Nowadays the world has entered the big data era. big data processing platforms, such as Hadoop and Spark, are increasingly adopted by many applications, in which there are numerous parameters that can be tuned to improve processing performance for big data platform operators. However, due to the large number of these parameters and the complex relationship among them, it is very time-consuming to manually tune parameters. Therefore, it is a challenge to automatically configure parameters as quickly as possible to optimize the performance of the current job. Existing auto-tuning methods often take a certain time before job runs to get the optimal configuration, which would increase the job's total processing time and reduce the overall efficiency of cluster. In this paper, we propose an adaptive tuning framework, mrMoulder, to recommend a near-optimal configuration for the new job in a short time. mrMoulder sets a self-extending configuration repository and a collaborative filtering based recommendation engine, to speed up the process of optimizing parameter configuration. We have deployed mrMoulder in a Hadoop cluster, and the experiment results have demonstrated that, for a new big data application, the recommend time of mrMoulder is only 20% to 30% of that for the existing auto-tuning methods, while the recommendation quality remains almost unchanged. (C) 2018 Elsevier B.V. All rights reserved.

关键词： big data processing Performance optimization Parameter tuning Online configuration recommendation Collaborative filtering

来源：评论

学校读者我要写书评

暂无评论

Cost-Aware big data processing Across Geo-Distributed datacenters

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2017年第11期28卷 3114-3127页

作者： Xiao, Wenhua Bao, Weidong Zhu, Xiaomin Liu, Ling Natl Univ Def Technol Sci & Technol Informat Syst Engn Lab Changsha 410073 Hunan Peoples R China Georgia Inst Technol Coll Comp 266 Ferst Dr Atlanta GA 30332 USA

With the globalization of service, organizations continuously produce large volumes of data that need to be analysed over geo-dispersed locations. Traditionally central approach that moving all data to a single cluster is inefficient or infeasible due to the limitations such as the scarcity of wide-area bandwidth and the low latency requirement of data processing. processing big data across geo-distributed datacenters continues to gain popularity in recent years. However, managing distributed MapReduce computations across geo-distributed datacenters poses a number of technical challenges: how to allocate data among a selection of geo-distributed datacenters to reduce the communication cost, how to determine the Virtual Machine (VM) provisioning strategy that offers high performance and low cost, and what criteria should be used to select a datacenter as the final reducer for big data analytics jobs. In this paper, these challenges is addressed by balancing bandwidth cost, storage cost, computing cost, migration cost, and latency cost, between the two MapReduce phases across datacenters. We formulate this complex cost optimization problem for data movement, resource provisioning and reducer selection into a joint stochastic integer nonlinear optimization problem by minimizing the five cost factors simultaneously. The Lyapunov framework is integrated into our study and an efficient online algorithm that is able to minimize the long-term time-averaged operation cost is further designed. Theoretical analysis shows that our online algorithm can provide a near optimum solution with a provable gap and can guarantee that the data processing can be completed within pre-defined bounded delays. Experiments on WorldCup98 web site trace validate the theoretical analysis results and demonstrate that our approach is close to the offline-optimum performance and superior to some representative approaches.

关键词： big data processing cloud computing data movement virtual machine scheduling online algorithm

来源：评论

学校读者我要写书评

暂无评论

BDEv 3.0: Energy efficiency and microarchitectural characterization of big data processing frameworks

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2018年 86卷 565-581页

作者： Veiga, Jorge Enes, Jonatan Exposito, Roberto R. Tourino, Juan Univ A Coruna Comp Architecture Grp Campus A Coruna La Coruna 15071 Spain

As the size of big data workloads keeps increasing, the evaluation of distributed frameworks becomes a crucial task in order to identify potential performance bottlenecks that may delay the processing of large datasets. While most of the existing works generally focus only on execution time and resource utilization, analyzing other important metrics is key to fully understanding the behavior of these frameworks. For example, microarchitecture-level events can bring meaningful insights to characterize the interaction between frameworks and hardware. Moreover, energy consumption is also gaining increasing attention as systems scale to thousands of cores. This work discusses the current state of the art in evaluating distributed processing frameworks, while extending our big data Evaluator tool (BDEv) to extract energy efficiency and microarchitecture-level metrics from the execution of representative big data workloads. An experimental evaluation using BDEv demonstrates its usefulness to bring meaningful information from popular frameworks such as Hadoop, Spark and Flink. (C) 2018 Elsevier B.V. All rights reserved.

关键词： big data processing Performance evaluation Energy efficiency Microarchitectural characterization

来源：评论

学校读者我要写书评

暂无评论

Enhancement of Infrared Images Using Super Resolution Techniques Based on big data processing

引用

MULTIMEDIA TOOLS AND APPLICATIONS 2020年第9-10期79卷 5671-5692页

作者： Abd El-Samie, Fathi E. Ashiba, Huda I. Shendy, H. Mansour, Hala M. Ahmed, Hossameldin M. Taha, Taha E. Dessouky, Moawad I. Elkordy, Mohamed F. Abd-Elnaby, Mohammed El-Fishawy, Adel S. Menoufia Univ Dept Elect & Elect Commun Engn Fac Elect Engn Menoufia 32952 Egypt Bilbis Higher Inst Engn Dept Elect & Elect Commun Bilbis Sharqia Egypt Benha Univ Shoubra Fac Engn Dept Elect & Elect Commun Banha Egypt Taif Univ Dept Comp Engn Coll Comp & Informat Technol Al Hawiya 21974 Saudi Arabia

This paper presents a super-resolution (SR) technique for enhancement of infrared (IR) images. The suggested technique relies on the image acquisition model, which benefits from the sparse representations of low-resolution (LR) and high-resolution (HR) patches of the IR images. It uses bicubic interpolation and minimum mean square error (MMSE) estimation in the prediction of the HR image with a scheme that can be interpreted as a feed-forward neural network. The suggested algorithm to overcome the problem of having only LR images due to hardware limitations is represented with a big data processing model. The performance of the suggested technique is compared with that of the standard regularized image interpolation technique as well as an adaptive block-by-block least-squares (LS) interpolation technique from the peak signal-to-noise ratio (PSNR) perspective. Numerical results reveal the superiority of the proposed SR technique.

关键词： Single-image super-resolution Image interpolation IR images Neural networks big data processing

来源：评论

学校读者我要写书评

暂无评论

Resource and Cost Aware Glowworm Mapreduce Optimization Based big data processing in Geo Distributed data Center

引用

WIRELESS PERSONAL COMMUNICATIONS 2021年第4期117卷 2831-2852页

作者： Nithyanantham, S. Singaravel, G. AL Amen Engn Coll Comp Sci & Engn Erode 638104 Tamil Nadu India KSR Coll Engn Autonomous Dept Informat Technol Tiruchengode 637215 Tamil Nadu India

Handling large data in geographically distributed information centers with resource and cost optimization is a key challenge. With several approaches being designed, handling a large volume of data in multiple datacenters in an inappropriate manner yet is considered to be a time-consuming process. To address these issues, a Multivariate Metaphor based Metaheuristic Glowworm Swarm Map-Reduce Optimization (MM-MGSMO) technique is presented. Here, with search space and large data volume as input for geo-distributed datacenters, glowworm (i.e. virtual machine) population is initialized. With each glowworm possessing a certain amount of luciferin (i.e. objective function), multiple objective functions (i.e. bandwidth, storage capacity, energy and computation cost) are defined for each virtual machine. Next, the glowworm position is updated according to the neighboring factor by means of probability. Followed by this, MapReduce function identifies the optimal virtual machine and accordingly allocation is performed, therefore improving data allocation efficiency. Besides, the workload is assigned across datacenters, reduction in computation cost and storage capacity is guaranteed. Experimental evaluation of MM-MGSMO approach with existing methods attained improved performances with factors such as data allocation efficiency, false-positive rate, storage capacity compared with other cutting edge technologies such as Joint optimization algorithm and Game theory-based dynamic resource allocation strategy.

关键词： big data processing Geo-distributed data center Glowworm Swarm optimization MapReduce function

来源：评论

学校读者我要写书评

暂无评论

An optimal big data processing for smart grid based on hybrid MDM/R architecture to strengthening RE integration and EE in datacenter

引用

JOURNAL OF AMBIENT INTELLIGENCE AND HUMANIZED COMPUTING 2019年第9期10卷 3709-3722页

作者： Mehenni, Abdeslam Alimazighi, Zaia Bouktir, Tarek Ahmed-Nacer, Mohamed Univ Sci Technol Houari Boumediene Algiers Algeria Univ Ferhat Abbas Setif 1 Setif Algeria

Supply chain is a hard business area, where you need to have a perfect balance between demand and supply, day in and day out, by an intricate system that sits underneath it all. Achieving such a system by meeting the ambitious targets of the agreement on climate change can only be achieved through an effective combination of energy efficiency and renewable energy integration. The uncertainty and variability of renewable energy generation can pose challenges for grid operators and can requires additional actions to balance the system. Significant researches which aim is to improve energy efficiency of data center indicates that operating reserves could be procured from many complex and costly techniques. In this paper, we investigate the problem from scheduling of workloads in a data center in order to minimize its energy consumption budget, minimize the conventional grid dependence, and maximize the renewable energy provided to data center, by the ability to temporarily delay or degrade service, with a modified supply-following algorithm. This algorithm attempts to align power consumption with the amount of wind power available, while minimizing the time by which jobs exceed their deadlines. Modification of the algorithm has been performed in the direction of big data processing (wind trace, workload requests, prices, horizontal ellipsis ), servers management. This modification is performed by jobs classification into predefined classes using the classification and regression trees algorithm. New hybrid architecture that manages the Meter data Management Repository MDM/R was introduced using MapReduce programming model for ETL process and Massive Parallel processing database for requests which strongly influences the accuracy and the speediness of the scheduler.

关键词： big data processing ETL Renewable energy integration Job scheduler Server state management

来源：评论

学校读者我要写书评

暂无评论

HDM: A Composable Framework for big data processing

引用

IEEE TRANSACTIONS ON big data 2018年第2期4卷 150-163页

作者： Wu, Dongyao Zhu, Liming Lu, Qinghua Sakr, Sherif CSIRO Data61 Sydney NSW 1670 Australia Univ New South Wales Sch Comp Sci & Engn Sydney NSW 2052 Australia China Univ Petr Coll Comp & Commun Engn Qingdao 266580 Peoples R China King Saud bin Abdulaziz Univ Hlth Sci Natl Guard Riyadh 14611 Saudi Arabia

Over the past years, frameworks such as MapReduce and Spark have been introduced to ease the task of developing big data programs and applications. However, the jobs in these frameworks are roughly defined and packaged as executable jars without any functionality being exposed or described. This means that deployed jobs are not natively composable and reusable for subsequent development. Besides, it also hampers the ability for applying optimizations on the data flow of job sequences and pipelines. In this paper, we present the Hierarchically Distributed data Matrix (HDM) which is a functional, strongly-typed data representation for writing composable big data applications. Along with HDM, a runtime framework is provided to support the execution, integration and management of HDM applications on distributed infrastructures. Based on the functional data dependency graph of HDM, multiple optimizations are applied to improve the performance of executing HDM jobs. The experimental results show that our optimizations can achieve improvements between 10 to 40 percent of the Job-Completion-Time for different types of applications when compared with the current state of art, Apache Spark.

关键词： big data processing parallel programming functional programming distributed systems system architecture

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：