检索结果-内蒙古大学图书馆

A multi-factor monitoring fault tolerance model based on a GPU cluster for big data processing

INFORMATION SCIENCES 2019年 496卷 300-316页

作者： Fang, Yuling Chen, Qingkui Xiong, Naixue Univ Shanghai Sci & Technol Shanghai 200093 Peoples R China Northeastern State Univ Dept Math & Comp Sci Tahlequah OK 74464 USA

High-performance computing clusters are widely used in large-scale data mining applications, and have higher requirements for persistence, stability and real-time use and sre therefore computationally intensive. To support large-scale data processing, we design a multi-factor real-time monitoring fault tolerance (MRMFT) model based on a GPU cluster. However, the higher clock frequency of GPU chips results in excessively high energy consumption in computing systems. Moreover, the ability to support a long-lasting high temperature operation varies greatly between different GPUs owing to the individual differences between the chips. In this paper, we design a GPU cluster energy consumption monitoring system based on wireless sensor networks (WSNs) and propose an energy consumption aware checkpointing (ECAC) for high energy consumption problems with the following two advantages: the system sets checkpoints according to actual energy consumption and the device temperature to improve the utilization of checkpoints and reduce time cost;and it exploits the parallel computing features of CPU and GPU to hide the CPU detection overhead in GPU parallel computation, and further reduce the time and energy consumption overhead in the fault tolerance phase. Using ECAC as the constraint and aiming for a persistent and reliable operation, the dynamic task migration mechanism is designed, and the reliability of the cluster is greatly improved. The theoretical analysis and experiment results show that the model improves the persistence and stability of the computing system while reducing checkpoint overhead. (C) 2018 Elsevier Inc. All rights reserved.

关键词： big data processing GPU cluster Persistence computing Energy consumption Fault tolerance Energy consumption aware checkpointing Task migration

来源：评论

学校读者我要写书评

暂无评论

FvRS: Efficiently identifying performance-critical data for improving performance of big data processing

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2019年 91卷 157-166页

作者： Xu, Gaoxiang Tan, Zhipeng Feng, Dan Yang, Laurence T. Zhou, Wei Zhang, Xinyan Zhang, Yang Xu, Jie Huazhong Univ Sci & Technol Wuhan Natl Lab Optoelect Key Lab Informat Storage Syst Sch Comp Sci & TechnolMinist Educ China Wuhan Hubei Peoples R China St Francis Xavier Univ Dept Comp Sci Antigonish NS Canada

Hybrid storage is widely implemented in big data processing by providing large storage capacity and high access speed in an economical manner. Performance-critical data are usually stored in SSD to obtain the most performance benefits with the least storage cost. Conventional scheme identifies performance critical data based on data's access hotness. But it does not consider data's I/O cost and may store low-cost data in SSD, resulting in the waste of SSD. The recently proposed scheme determines performance-critical data based on both access hotness and I/O cost. However, it fails to accurately evaluate I/O cost, thus still distributes many low-cost data on SSD. In this paper, we propose a sequentiality-aware identification scheme for performance-critical data, called FvRS, which boots the accuracy of I/O cost evaluation by exploiting data's access sequentiality. The key idea is to evaluate data's I/O cost based on both request size and access sequentiality. By properly identifying high-cost hot data, FvRS maximizes the utilization of SSD to improve system performance. In addition, FvRS maintains performance-critical data in a real-time table to reduce the identification overhead. We have implemented FvRS in a hybrid storage system in Linux. Extensive evaluations using three real-workload traces and a famous benchmark Postmark demonstrate the accuracy and efficiency of FvRS. Compared with the state-of-the-art schemes, such as hotness-based identification and cost-based identification, FvRS reduces I/O response time by 10.3%similar to 45.6% and 16.3%similar to 25.1%, respectively. (C) 2018 Elsevier B.V. All rights reserved.

关键词： big data processing Performance-critical data I/O cost evaluation Access sequentiality Hybrid storage

来源：评论

学校读者我要写书评

暂无评论

data Prefetching and Eviction Mechanisms of In-Memory Storage Systems Based on Scheduling for big data processing

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2019年第8期30卷 1738-1752页

作者： Chen, Chien-Hung Hsia, Ting-Yuan Huang, Yennun Kuo, Sy-Yen Natl Taiwan Univ Dept Elect Engn Taipei 10617 Taiwan Acad Sinica Res Ctr Informat Technol Innovat Taipei 11529 Taiwan

In-memory techniques keep data into faster and more expensive storage media for improving performance of big data processing. However, existing mechanisms do not consider how to expedite the data processing applications that access the input datasets only once. Another problem is how to reclaim memory without affecting other running applications. In this paper, we provide scheduling-aware data prefetching and eviction mechanisms based on Spark, Alluxio, and Hadoop. The mechanisms prefetch data and release memory resources based on the scheduling information. A mathematical method is proposed for maximizing the reduction of data access time. To make the mechanisms applicable in large-scale environments, we propose a heuristic algorithm to reduce the computational time. Furthermore, an enhanced version of the heuristic algorithm is also proposed to increase the amount of prefetched data. Finally, we perform real-testbed and simulation experiments to show the effectiveness of the proposed mechanisms.

关键词： big data processing in-memory systems scheduling information data prefetching data eviction

来源：评论

学校读者我要写书评

暂无评论

mrMoulder: A recommendation-based adaptive parameter tuning approach for big data processing platform

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2019年 93卷 570-582页

作者： Cai, Lin Qi, Yong Wei, Wei Wu, Jinsong Li, Jingwei Xi An Jiao Tong Univ Sch Elect & Informat Engn Xian 710049 Shaanxi Peoples R China Xian Univ Technol Sch Comp Sci & Engn Xian 710048 Shaanxi Peoples R China Univ Chile Dept Elect Engn Av Tupper 2007 Santiago 8370451 Chile

Nowadays the world has entered the big data era. big data processing platforms, such as Hadoop and Spark, are increasingly adopted by many applications, in which there are numerous parameters that can be tuned to improve processing performance for big data platform operators. However, due to the large number of these parameters and the complex relationship among them, it is very time-consuming to manually tune parameters. Therefore, it is a challenge to automatically configure parameters as quickly as possible to optimize the performance of the current job. Existing auto-tuning methods often take a certain time before job runs to get the optimal configuration, which would increase the job's total processing time and reduce the overall efficiency of cluster. In this paper, we propose an adaptive tuning framework, mrMoulder, to recommend a near-optimal configuration for the new job in a short time. mrMoulder sets a self-extending configuration repository and a collaborative filtering based recommendation engine, to speed up the process of optimizing parameter configuration. We have deployed mrMoulder in a Hadoop cluster, and the experiment results have demonstrated that, for a new big data application, the recommend time of mrMoulder is only 20% to 30% of that for the existing auto-tuning methods, while the recommendation quality remains almost unchanged. (C) 2018 Elsevier B.V. All rights reserved.

关键词： big data processing Performance optimization Parameter tuning Online configuration recommendation Collaborative filtering

来源：评论

学校读者我要写书评

暂无评论

An optimal big data processing for smart grid based on hybrid MDM/R architecture to strengthening RE integration and EE in datacenter

引用

JOURNAL OF AMBIENT INTELLIGENCE AND HUMANIZED COMPUTING 2019年第9期10卷 3709-3722页

作者： Mehenni, Abdeslam Alimazighi, Zaia Bouktir, Tarek Ahmed-Nacer, Mohamed Univ Sci Technol Houari Boumediene Algiers Algeria Univ Ferhat Abbas Setif 1 Setif Algeria

Supply chain is a hard business area, where you need to have a perfect balance between demand and supply, day in and day out, by an intricate system that sits underneath it all. Achieving such a system by meeting the ambitious targets of the agreement on climate change can only be achieved through an effective combination of energy efficiency and renewable energy integration. The uncertainty and variability of renewable energy generation can pose challenges for grid operators and can requires additional actions to balance the system. Significant researches which aim is to improve energy efficiency of data center indicates that operating reserves could be procured from many complex and costly techniques. In this paper, we investigate the problem from scheduling of workloads in a data center in order to minimize its energy consumption budget, minimize the conventional grid dependence, and maximize the renewable energy provided to data center, by the ability to temporarily delay or degrade service, with a modified supply-following algorithm. This algorithm attempts to align power consumption with the amount of wind power available, while minimizing the time by which jobs exceed their deadlines. Modification of the algorithm has been performed in the direction of big data processing (wind trace, workload requests, prices, horizontal ellipsis ), servers management. This modification is performed by jobs classification into predefined classes using the classification and regression trees algorithm. New hybrid architecture that manages the Meter data Management Repository MDM/R was introduced using MapReduce programming model for ETL process and Massive Parallel processing database for requests which strongly influences the accuracy and the speediness of the scheduler.

关键词： big data processing ETL Renewable energy integration Job scheduler Server state management

来源：评论

学校读者我要写书评

暂无评论

Real-time big data processing for anomaly detection: A Survey

引用

INTERNATIONAL JOURNAL OF INFORMATION MANAGEMENT 2019年 45卷 289-307页

作者： Habeeb, Riyaz Ahamed Ariyaluran Nasaruddin, Fariza Gani, Abdullah Hashem, Ibrahim Abaker Targio Ahmed, Ejaz Imran, Muhammad Univ Malaya Fac Comp Sci & Informat Technol Dept Informat Syst Kuala Lumpur 50603 Malaysia Taylors Univ Sch Comp & Informat Technol Subang Jaya 47500 Selangor Malaysia Univ Malaya Ctr Res Mobile Cloud Comp Kuala Lumpur Malaysia King Saud Univ Coll Appl Comp Sci Riyadh Saudi Arabia

The advent of connected devices and omnipresence of Internet have paved way for intruders to attack networks, which leads to cyber-attack, financial loss, information theft in healthcare, and cyber war. Hence, network security analytics has become an important area of concern and has gained intensive attention among researchers, off late, specifically in the domain of anomaly detection in network, which is considered crucial for network security. However, preliminary investigations have revealed that the existing approaches to detect anomalies in network are not effective enough, particularly to detect them in real time. The reason for the inefficacy of current approaches is mainly due the amassment of massive volumes of data though the connected devices. Therefore, it is crucial to propose a framework that effectively handles real time big data processing and detect anomalies in networks. In this regard, this paper attempts to address the issue of detecting anomalies in real time. Respectively, this paper has surveyed the state-of-the-art real-time big data processing technologies related to anomaly detection and the vital characteristics of associated machine learning algorithms. This paper begins with the explanation of essential contexts and taxonomy of real-time big data processing, anomalous detection, and machine learning algorithms, followed by the review of big data processing technologies. Finally, the identified research challenges of real-time big data processing in anomaly detection are discussed.

关键词： Real-time big data processing Anomaly detection and machine learning algorithms

来源：评论

学校读者我要写书评

暂无评论

A Distributed Multi-Agent System (MAS) Application For continuous and Integrated big data processing 15th

A Distributed Multi-Agent System (MAS) Application For conti...

引用

15th European Conference on Ambient Intelligence (AmI)

作者： Shashaj, Ariona Mastrorilli, Federico Morrelli, Massimiliano Pansini, Giacomo Iannucci, Enrico Polito, Massimiliano Network Contacts Molfetta BA Italy

ISBN: (纸本)9783030342555;9783030342548

With the advent of Ambient Intelligence (AmI) as an interdisciplinary methodology which ranges from Ubiquitous and Pervasive computing to Artificial Intelligence with the final aim to create a sensitive and responsive environment, the focus is moving towards integrated solutions of the encompassed technologies. Multi-Agent System (MAS) approach, characterized by a set of autonomous intelligent agents which can cooperate in order to achieve a common goal, can help the development of AmI integrated solutions. In this paper, we present a distributed MAS environment, Multi-Agent Specialized system (MASs), which supports the development of integrated AmI solutions. An application scenario considering the case of continuous big data processing is shown.

关键词： Multi-Agent system Distributed system big data processing

来源：评论

学校读者我要写书评

暂无评论

Fault tolerance in big data storage and processing systems: A review on challenges and solutions

引用

AIN SHAMS ENGINEERING JOURNAL 2022年第2期13卷 101538-101538页

作者： Saadoon, Muntadher Ab Hamid, Siti Hafizah Sofian, Hazrina Altarturi, Hamza H. M. Azizul, Zati Hakim Nasuha, Nur Univ Malaya Fac Comp Sci & Informat Technol Dept Software Engn Kuala Lumpur 50603 Malaysia

big data systems are sufficiently stable to store and process a massive volume of rapidly changing data. However, big data systems are composed of large-scale hardware resources that make their subspecies easily fail. Fault tolerance is the main property of such systems because it maintains availability, reliability, and constant performance during faults. Achieving an efficient fault tolerance solution in a big data system is challenging because fault tolerance must meet some constraints related to the system performance and resource consumption. This study aims to provide a consistent understanding of fault tolerance in big data systems and highlights common challenges that hinder the improvement in fault tolerance efficiency. The fault tolerance solutions applied by previous studies intended to address the identified challenges are reviewed. The paper also presents a perceptive discussion of the findings derived from previous studies and proposes a list of future directions to address the fault tolerance challenges. (C) 2021 THE AUTHORS. Published by Elsevier BV on behalf of Faculty of Engineering, Ain Shams University.

关键词： Fault tolerance Fault detection Fault recovery big data storage big data processing

来源：评论

学校读者我要写书评

暂无评论

Transformation-Based Streaming Workflow Allocation on Geo-Distributed datacenters for Streaming big data processing

引用

IEEE TRANSACTIONS ON SERVICES COMPUTING 2019年第4期12卷 654-668页

作者： Chen, Wuhui Paik, Incheon Hung, Patrick C. K. Univ Aizu Sch Comp Sci & Engn Aizu Wakamatsu Fukushima 9658580 Japan Univ Ontario Inst Technol Fac Business & IT Oshawa ON L1H 7K4 Canada Natl Taipei Univ Technol Dept Elect Engn Taipei 10608 Taiwan

The cost-minimization problem for streaming workflow (SW) has already become increasingly important and even critical in stream big data processing, particularly for geographically distributed datacenters, because of its huge demand on computing and communicating resources. Existing virtual machine (VM) allocation algorithms in cloud computing have been widely applied to batch-processing models;however, none of them can be successfully applied to SW because: 1) they failed to adapt the continuous execution characteristic of SW;and 2) most of them are all based on the assumption that the price of traffic and VMs among datacenters are uniform. In this paper, we propose a transformation-based SW allocation algorithm with the goal of cost-minimization for stream big data processing in geographically distributed datacenters, considering the characteristics of SW and price heterogeneity among geographically distributed datacenters. We first propose a cost-aware workflow transformation framework based on eight well-designed and verified transformation rules for cost reduction to adapt the continuous execution characteristic of SW. We then formulate the joint VM-traffic optimization problem and show that it is NP-hard. To produce the optimal solution in polynomial time, we then transform the SW allocation problem into the minimum-cost maximum-flow problem, considering both traffic and VMs price heterogeneity. Finally, our experimental results validate the high cost efficiency of our approach with lower computing and communicating costs by optimizing the workflow specification and joint VM-traffic cost optimization.

关键词： big data processing stream data workflow transformation geo-distributed datacenters workflow allocation

来源：评论

学校读者我要写书评

暂无评论

Research and Application of Spark Platform on big data processing in Intelligent Agriculture of Jilin Province 1

引用

11th IFIP WG 5.14 International Conference on Computer and Computing Technologies in Agriculture (CCTA)

作者： Fu, Siwei Chen, Guifen Zhao, Shan Xiao, Enze Jilin Agr Univ Changchun 130118 Peoples R China

ISBN: (数字)9783030061791

ISBN: (纸本)9783030061791;9783030061784

Aiming at the demand of real-time massive data processing of Intelligent Agriculture in Jilin Province, this paper studies the big data processing of Intelligent Agriculture in Jilin Province based on Spark platform by acquiring real-time data through monitoring platform. This study first conducted the performance comparison experiment of Hadoop and Spark data processing platform, then used the Spark distributed cluster computing platform, real-time processing the big data of monitoring area. The experimental results show that the Spark platform speeds up 11.4 times faster than the Hadoop platform in the case of 100 million data sizes;and based on the Spark platform for real-time processing of big data intelligent agricultural monitoring network, not only provides memory calculations to reduce IO overhead, but also the results are faster and more accurate. The research results provide strong support for the implementation of precision agriculture technology in intelligent agriculture.

关键词： Spark big data processing MapReduce Intelligent Agriculture in Jilin Province

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：