Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark

作     者：Mavridis, Ilias Karatza, Helen 

作者机构：Aristotle Univ Thessaloniki Dept Informat Thessaloniki 54124 Greece 

出 版 物：《JOURNAL OF SYSTEMS AND SOFTWARE》 (系统与软件杂志)

年 卷 期：2017年第125卷

页      面：133-151页

核心收录：

学科分类：08[工学] 0835[工学-软件工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

主　　题：Log Analysis Cloud Apache Hadoop Apache Spark Performance Evaluation 

摘      要：Log files are generated in many different formats by a plethora of devices and software. The proper analysis of these files can lead to useful information about various aspects of each system. Cloud computing appears to be suitable for this type of analysis, as it is capable to manage the high production rate, the large size and the diversity of log files. In this paper we investigated log file analysis with the cloud computational frameworks Apache (TM) Hadoop (R) and Apache Spark (TM). We developed realistic log file analysis applications in both frameworks and we performed SQL-type queries in real Apache Web Server log files. Various experiments were performed with different parameters in order to study and compare the performance of the two frameworks. (C) 2016 Elsevier Inc. All rights reserved.