检索结果-内蒙古大学图书馆

Performance evaluation of cloud-based log file analysis with apache hadoop and apache Spark

JOURNAL OF SYSTEMS AND SOFTWARE 2017年 125卷 133-151页

作者： Mavridis, Ilias Karatza, Helen Aristotle Univ Thessaloniki Dept Informat Thessaloniki 54124 Greece

Log files are generated in many different formats by a plethora of devices and software. The proper analysis of these files can lead to useful information about various aspects of each system. Cloud computing appears to be suitable for this type of analysis, as it is capable to manage the high production rate, the large size and the diversity of log files. In this paper we investigated log file analysis with the cloud computational frameworks apache (TM) hadoop (R) and apache Spark (TM). We developed realistic log file analysis applications in both frameworks and we performed SQL-type queries in real apache Web Server log files. Various experiments were performed with different parameters in order to study and compare the performance of the two frameworks. (C) 2016 Elsevier Inc. All rights reserved.

关键词： Log Analysis Cloud apache hadoop apache Spark Performance Evaluation

来源：评论

学校读者我要写书评

暂无评论

Developing a big data analytics platform using apache hadoop Ecosystem for delivering big data services in libraries

引用

DIGITAL LIBRARY PERSPECTIVES 2024年第2期40卷 160-186页

作者： Singh, Ranjeet Kumar Indian Stat Inst Documentat Res & Training Ctr Bangalore Ctr Bengaluru India Univ Calcutta Dept Lib & Informat Sci Kolkata India

PurposeAlthough the challenges associated with big data are increasing, the question of the most suitable big data analytics (BDA) platform in libraries is always significant. The purpose of this study is to propose a solution to this ***/methodology/approachThe current study identifies relevant literature and provides a review of big data adoption in libraries. It also presents a step-by-step guide for the development of a BDA platform using the apache hadoop Ecosystem. To test the system, an analysis of library big data using apache Pig, which is a tool from the apache hadoop Ecosystem, was performed. It establishes the effectiveness of apache hadoop Ecosystem as a powerful BDA solution in *** can be inferred from the literature that libraries and librarians have not taken the possibility of big data services in libraries very seriously. Also, the literature suggests that there is no significant effort made to establish any BDA architecture in libraries. This study establishes the apache hadoop Ecosystem as a possible solution for delivering BDA services in *** limitations/implicationsThe present work suggests adapting the idea of providing various big data services in a library by developing a BDA platform, for instance, providing assistance to the researchers in understanding the big data, cleaning and curation of big data by skilled and experienced data managers and providing the infrastructural support to store, process, manage, analyze and visualize the big *** implicationsThe study concludes that apache hadoops' hadoop Distributed File System and MapReduce components significantly reduce the complexities of big data storage and processing, respectively, and apache Pig, using Pig Latin scripting language, is very efficient in processing big data and responding to queries with a quick response ***/valueAccording to the study, there are significantly fewer efforts made to analyze big data from l

关键词： Big data Library big data analytics apache hadoop apache Pig Data analysis Pig Latin

来源：评论

学校读者我要写书评

暂无评论

A comprehensive bibliometric analysis of apache hadoop from 2008 to 2020

引用

International Journal of Intelligent Computing and Cybernetics 2023年第1期16卷 99-120页

作者： Jianpeng Zhang Mingwei Lin Fujian Normal University FuzhouChina

Purpose-The purpose of this paper is to make an overview of 6,618 publications of apache hadoop from 2008 to 2020 in order to provide a conclusive and comprehensive analysis for researchers in this field,as well as a preliminary knowledge of apache hadoop for interested ***/methodology/approach-This paper employs the bibliometric analysis and visual analysis approaches to systematically study and analyze publications about apache hadoop in the Web of Science *** study aims to investigate the topic of apache hadoop by means of bibliometric analysis with the aid of visualization *** the bibliometric analysis of the collected documents,this paper analyzes the main statistical characteristics and cooperation *** themes,research hotspots and future development trends are also investigated through the keyword ***-The research on apache hadoop is still the top priority in the future,and how to improve the performance of apache hadoop in the era of big data is one of the research *** limitations/implications-This paper makes a comprehensive analysis of apache hadoop with methods of bibliometrics,and it is valuable for researchers can quickly grasp the hot topics in this ***/value-This paper draws the structural characteristics of the publications in this field and summarizes the research hotspots and trends in this field in recent years,aiming to understand the development status and trends in this field and inspire new ideas for researchers.

关键词： apache hadoop Big data Bibliometric analysis Hot topic Development trends

来源：评论

学校读者我要写书评

暂无评论

Processing LIDAR Data with apache hadoop

Processing LIDAR Data with Apache Hadoop

引用

GIS Symposium on the Rise of Big Spatial Data

作者： Ruzicka, Jan Orcik, Lukas Ruzickova, Katerina Kisztner, Juraj VSB Tech Univ Ostrava Fac Min & Geol Inst Geoinformat 17 Listopadu 15-2172 Ostrava 70833 Czech Republic VSB Tech Univ Ostrava Fac Elect Engn & Comp Sci Dept Telecommun 17 Listopadu 15 Ostrava 70833 Czech Republic VSB Tech Univ Ostrava Inst Geol Engn 17 Listopadu 15 Ostrava 70833 Czech Republic

ISBN: (纸本)9783319451237;9783319451220

The paper is focused on research in the area of processing LIDAR data with apache hadoop. Our team is managing an information system that is able to calculate probability of existence of different objects in space and time. The system works with a lot of different data sources, including large datasets. We may process LIDAR data in the future as well, so we were interested how to process LIDAR data with apache hadoop. Our colleagues from the institute of geology are using LIDAR data and have a lot of problems with their processing. The main problem is time that is necessary to process the data to build simple DEM in GRID format that is commonly used in the geology (geomorphology) area. We were interested if we could help them with processing such kind of data with MapReduce architecture on apache hadoop to produce GRID data suitable to their needs. The paper presents results of our research.

关键词： LIDAR MapReduce GRID apache hadoop Cloudera

来源：评论

学校读者我要写书评

暂无评论

Creating Large Size of Data with apache hadoop

Creating Large Size of Data with Apache Hadoop

引用

GIS Symposium on the Rise of Big Spatial Data

作者： Ruzicka, Jan Kocich, David Orcik, Lukas Svozilik, Vladislav VSB Tech Univ Ostrava Fac Min & Geol Inst Geoinformat 17 Listopadu 15-2172 Ostrava 70833 Czech Republic VSB Tech Univ Ostrava Fac Elect Engn & Comp Sci Dept Telecommun 17 Listopadu 15 Ostrava 70833 Czech Republic

ISBN: (纸本)9783319451237;9783319451220

The paper is focused on research in the area of building large datasets using apache hadoop. Our team is managing an information system that is able to calculate probability of existence of different objects in space and time. The system works with a lot of different data sources, including large datasets. The workflow of data processing is quite complicated and time consuming, so we were looking for some framework that could help with system management and, if possible, to speed up data processing as well. apache hadoop was selected as a platform for enhance our information system. apache hadoop is usually used for processing large datasets, but in a case of our information scystem is necessary to perform other types of tasks as well. The systems computes spatio-temporal relations between different types of objects. This means that from relatively small amount of records (thousands) are built relatively large datasets (millions of records). For this purposes is usually used PostgreSQL/PostGIS database or tools written in Java or other language. Our research was focused to determination if we could simply move some of this tasks to apache hadoop platform using simple SQL editor like Hive. We have selected two types of common tasks and tested them on PostgreSQL and apache hadoop (Hive) platform to be able compare time necessary to complete these tasks. The paper presents results of our research.

关键词： Data creating apache hadoop Cloudera Hive PostgreSQL

来源：评论

学校读者我要写书评

暂无评论

Theoretical and Empirical Comparison of Big Data Image Processing with apache hadoop and Sun Grid Engine

Theoretical and Empirical Comparison of Big Data Image Proce...

引用

Conference on Medical Imaging - Imaging Informatics for Healthcare, Research, and Applications

作者： Bao, Shunxing Weitendorf, Frederick D. Plassard, Andrew J. Huo, Yuankai Gokhale, Aniruddha Landman, Bennett A. Vanderbilt Univ Comp Sci 221 Kirkland Hall Nashville TN 37235 USA Vanderbilt Univ Elect Engn Nashville TN 37235 USA

ISBN: (纸本)9781510607217;9781510607224

The field of big data is generally concerned with the scale of processing at which traditional computational paradigms break down. In medical imaging, traditional large scale processing uses a cluster computer that combines a group of workstation nodes into a functional unit that is controlled by a job scheduler. Typically, a shared-storage network file system (NFS) is used to host imaging data. However, data transfer from storage to processing nodes can saturate network bandwidth when data is frequently uploaded/ retrieved from the NFS, e.g., "short" processing times and/or "large" datasets. Recently, an alternative approach using hadoop and HBase was presented for medical imaging to enable co-location of data storage and computation while minimizing data transfer. The benefits of using such a framework must be formally evaluated against a traditional approach to characterize the point at which simply "large scale" processing transitions into "big data" and necessitates alternative computational frameworks. The proposed hadoop system was implemented on a production lab-cluster alongside a standard Sun Grid Engine (SGE). Theoretical models for wall-clock time and resource time for both approaches are introduced and validated. To provide real example data, three T1 image archives were retrieved from a university secure, shared web database and used to empirically assess computational performance under three configurations of cluster hardware (using 72, 109, or 209 CPU cores) with differing job lengths. Empirical results match the theoretical models. Based on these data, a comparative analysis is presented for when the hadoop framework will be relevant and non-relevant for medical imaging.

关键词： apache hadoop Sun Grid Engine Verification

来源：评论

学校读者我要写书评

暂无评论

A Modern Data Architecture with apache hadoop 1

A Modern Data Architecture with Apache Hadoop

引用

International Conference on Green Computing and Internet of Things (ICGCIoT)

作者： Singh, Tripty Darshan, V. S. Amrita Sch Engn Dept CSE Bangalore Karnataka India

ISBN: (纸本)9781467379106

This paper represents the analysis of the existing architecture framework used across *** also emphasizes on the modern architecture in integration with apache hadoop. The existing data architecture is under pressure from new data and machine generated data for the upcoming years that is due to emergence of new data types there has been tremendous pressure on all the data systems within an enterprise over the years. An Exponential Growth has been estimated from 2.8 Zeta Byte of data in 2012 to grow to 40 Zeta Byte by 2020 and new data type growth is estimated to be eighty five percent with computer created information being expanded to 15 times all the more by 2020

关键词： Data Architecture apache hadoop Zeta Byte Map Reduce Big Data

来源：评论

学校读者我要写书评

暂无评论

Context-Aware Scheduling for apache hadoop over Pervasive Environments 6th

Context-Aware Scheduling for Apache Hadoop over Pervasive En...

引用

6th International Conference on Ambient Systems, Networks and Technologies (ANT) / 5th International Conference on Sustainable Energy Information Technology (SEIT)

作者： Cassales, Guilherme W. Charao, Andrea S. Pinheiro, Manuele Kirsch Souveyet, Carine Steffenel, Luiz A. Univ Fed Santa Maria Lab Sistemas Computacao BR-97119900 Santa Maria RS Brazil Univ Paris 01 Ctr Rech Informat F-75231 Paris 05 France Univ Reims Lab CReSTIC Equipe SysCom Reims France

This article proposes to improve apache hadoop scheduling through the usage of context-awareness. apache hadoop is the most popular implementation of the MapReduce paradigm for distributed computing, but its design doesn't adapt automatically to computing nodes' context and capabilities. By introducing context-awareness into hadoop, we intent to dynamically adapt its scheduling to the execution environment. This is a necessary feature in the context of pervasive grids, which are heterogeneous, dynamic and shared environments. The solution has been incorporated into hadoop and evaluated through controlled experiments. The experiments demonstrate that context-awareness provides comparative performance gains, especially when part of the resources disappear during execution. (C) 2015 The Authors. Published by Elsevier B.V.

关键词： Context-Awareness Map Reduce apache hadoop Task Scheduling Pervasive Grid Big Data

来源：评论

学校读者我要写书评

暂无评论

Processing of Big Educational Data in the Cloud Using apache hadoop

Processing of Big Educational Data in the Cloud Using Apache...

引用

International Conference on Information Society (i-Society)

作者： Machova, Renata Komarkova, Jitka Lnenicka, Martin Univ Pardubice Fac Econ & Adm Pardubice Czech Republic

ISBN: (纸本)9781908320612

The amount of available educational data is growing and the educational institutions try to organize and understand large complex datasets. Since these big data are an emerging field of study, there is no simple solution to process or analyze them. This paper aims to provide a model approach to process big educational data from the Moodle system in the cloud using the apache hadoop cluster. An experimental implementation of the proposed model approach is included to validate this solution. The results then suggest that the proposed model approach has a potential to be widely used for the processing of big educational data, especially with user interaction data.

关键词： Big educational data Cloud computing Moodle apache hadoop

来源：评论

学校读者我要写书评

暂无评论

On The Performance of apache hadoop in a Tiny Private IaaS Cloud

On The Performance of Apache Hadoop in a Tiny Private IaaS C...

引用

10th International Conference on Information Technology - New Generations (ITNG)

作者： Loewen, Gabriel Galloway, Michael Vrbsky, Susan Univ Alabama Dept Comp Sci Tuscaloosa AL 35487 USA

ISBN: (纸本)9780769549675

High performance and parallel computing are traditionally implemented on very large dedicated compute clusters. However, as many organizations begin to adopt service-oriented cloud-based infrastructures, we can expect to see the development of parallel computing in the cloud. The goal of a parallel compute cluster is to divide a large job into several small jobs, execute the small jobs in parallel on many compute nodes, and then combine the results in some coherent manner. The biggest hurdle in moving this type of service to a cloud-based infrastructure is that performance will undoubtedly be affected by many factors, particularly those related to virtualization in clouds, such as memory and CPU overhead, limited resources, and others relating to hardware virtualization. In order to fully understand how virtualization can affect parallel computing in a tiny private cloud, we have devised four case studies that examine the performance of apache hadoop in varying environments on our private cloud. Our case studies are comprised of a baseline or bare metal (non-virtualized) cluster deployment consisting of seven nodes, a seven-node virtual machine cluster, a twenty-node virtual machine cluster, and an optimized seven-node virtual machine cluster. Results show that, although small data sets result in comparable job completion times, as the data size increases the performance of apache hadoop is affected greatly by virtualization even when we attempt to optimize the configuration of our cloud.

关键词： Performance Eucalyptus Infrastructure-as-a-Service apache hadoop Parallel Computing Virtualization Case Studies Kernel-Based Virtual Machine Hypervisor

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：