检索结果-内蒙古大学图书馆

12th International Conference on Networking, Architecture, and Storage (NAS)

作者： Li, Zhaowei Yan, Yunlong Mo, Jintao Wen, Zhaocong Wu, Junmin Guangdong Power Grid Co Ltd Dongguan Power Supply Bur Guangzhou Guangdong Peoples R China Univ Sci & Technol China Suzhou Inst Adv Study Hefei Anhui Peoples R China

ISBN: (纸本)9781538634868

hadoop as an open source framework for dealing with Big Data can be processed to calculate large amounts of data in parallel, which has attracted more and more attention in academia and industry. This paper analyzes the methods of In-Memory file system using HDFS Lazy Persist strategy and Alluxio to upgrade system I/O efficiency. Besides, in order to avoid the problem that Lazy Persist strategy needs to be triggered manually each time, we propose HDFS Lazy Persist strategy automatic trigger mechanism based on the statistics of data access information.

关键词： In-Memory file system hadoop distributed file system Alluxio Big Data

来源：评论

学校读者我要写书评

暂无评论

Scalable Provenance Storage and Querying Using Pig Latin for Big Data Workflows 14

Scalable Provenance Storage and Querying Using Pig Latin for...

引用

IEEE International Conference on Services Computing (SCC)

作者： Bhuyan, Fahima Lu, Shiyong Ruan, Dong Zhang, Jia Wayne State Univ Dept Comp Sci Detroit MI 48202 USA Carnegie Mellon Univ Silicon Valley Elect & Comp Engn Mountain View CA USA

ISBN: (纸本)9781538620052

Provenance refers to the information about the derivation history of a data product. It is important for evaluating the quality and trustworthiness of a data product and ensuring the reproducibility of scientific discoveries. Much research has been done on storing and querying scientific workflow provenance - provenance that is produced in the execution of data-centric scientific workflows. To address the challenges of big data in increasing volume, velocity and variety, a new generation of scientific workflows, called big data workflows are under active research. As both data and workflows increase in their scale, the scale of provenance naturally increases, calling for a new scalable storage and querying infrastructure. This paper leverages Pig Latin, a high-level platform for creating programs that run on Apache hadoop, and OPQL, a graph-level provenance query language, to build a scalable provenance storage and querying system for big data workflows. Our main contributions are: i) we propose algorithms to translate OPQL constructs to equivalent Pig Latin programs;ii) we extend OPQL, to support the W3C PROV-DM standard provenance model;iii) we develop and evaluate our system on provenance datasets from the UTPB benchmark;and (iv) we create some visual OPQL constructs in the DATAVIEW big data workflow system to facilitate the easy creation of complex OPQL queries in a visual workflow style. Our preliminary experimental study shows the feasibility of our framework for big-data-scale provenance storage and querying.

关键词： Provenance Pig Big data hadoop distributed file system Query language

来源：评论

学校读者我要写书评

暂无评论

Analyzing Network Traffic Data Using Hive Queries

Analyzing Network Traffic Data Using Hive Queries

引用

IEEE SoutheastCon Conference

作者： Patel, Dharaben Yuan, Xiaohong Roy, Kaushik Abernathy, Aakiel NC A&T State Univ Dept Comp Sci Greensboro NC 27401 USA

ISBN: (纸本)9781538615393

Billions of devices are connected together with internet to serve the communication. Network monitoring to detect various security threats has become crucial in any organization. In this paper, we analyze large amount of network traffic data using Hive database in hadoop distributed file system (HDFS) environment. Hive queries are developed to identify security threats. The results of queries are demonstrated and the Hive Client application is developed where all the queries can be integrated. An Apache Zeppelin Visualization Tool is also introduced which can provide more insights on the dataset.

关键词： Network data security analysis hadoop distributed file system Visualization Tool Apache Zeppelin

来源：评论

学校读者我要写书评

暂无评论

Efficient Storage and Processing of Video Data for Moving Object Detection Using hadoop/MapReduce

引用

International Conference on Signal, Networks, Computing, and systems (ICSNCS)

作者： Parsola, Jyoti Gangodkar, Durgaprasad Mittal, Ankush Graph Era Univ Dept Comp Applicat Dehra Dun Uttar Pradesh India Graph Era Univ Dept Comp Sci & Engn Dehra Dun Uttar Pradesh India

ISBN: (纸本)9788132235927;9788132235903

Technological advances and easy availability of low cost video camera has further encouraged users for deploying network of surveillance systems. These systems generate massive data. Thus, storage and processing of the huge video data for application such as video forensics, post event investigation etc., has emerged as a challenging problem to the research community. In this paper we propose a powerful approach that makes use of hadoop distributed file system (HDFS) for efficient storage of video data and programming model called MapReduce for data intensive computing. The proposed approach detects moving objects and provides their coordinates which can be used for localizing post event investigation. We have analyzed the storage and processing of video data of varying resolution and size to assess the performance of proposed approach.

关键词： Motion detection hadoop distributed file system MapReduce

来源：评论

学校读者我要写书评

暂无评论

Addressing hadoop's Small file Problem With an Appendable Archive file Format 17

Addressing Hadoop's Small File Problem With an Appendable Ar...

引用

14th ACM International Conference on Computing Frontiers

作者： Renner, Thomas Mueller, Johannes Thamsen, Lauritz Kao, Odej Tech Univ Berlin Berlin Germany

ISBN: (纸本)9781450344876

hadoop has been used widely for data analytic tasks in various domains. At the same time, data volume is expected to grow even further in the next years. hadoop recently introduced the concept Archival Storage, an automated tiered storage technique for increasing storage capacity for long-term storage. However, hadoop distributed file system's scalability is limited by the total number of files that can be stored, and it is likely that the number of files increases fast when using it for archival purposes. This paper presents an approach for improving HDFS' scalability when using it as an archival storage. We present a tool that extends hadoop Archive to an appendable file format. New files are appended to one of the existing archive data files efficiently without rewriting the whole archive. Therefore, a first fit algorithm is used to fill up the often not fully utilized fixed-sized data blocks of the archive data files. Index files are updated using a red-black tree providing guaranteed fast lookup and insert performance. We show that the tool performs well for different sizes of archives and number of files to add. By distributing new files efficiently, we also reduce the number of data blocks needed for archiving and, thus, reduce the memory footprint on the NameNode.

关键词： file systems hadoop distributed file system HDFS Metadata Management Archival Storage

来源：评论

学校读者我要写书评

暂无评论

Concept Generation in Formal Concept Analysis Using MapReduce Framework 1

Concept Generation in Formal Concept Analysis Using MapReduc...

引用

International Conference on Big Data Analytics and Computational Intelligence (ICBDAC)

作者： Chunduri, Ragahvendra Kumar Cherukuri, Aswani Kumar Tamir, Mike VIT Univ Sch Informat Technol Vellore 632014 Tamil Nadu India Univ Calif Berkeley Sch Informat Berkeley CA 94720 USA

ISBN: (纸本)9781509064007

Searching for interesting patterns in large data sources plays a vital part in Data Science. There are several techniques, which can be applied to data sources to extract the patterns that are required. Formal Concept Analysis (FCA) is one such mathematical method for data analysis with flourishing popularity across distinct domains, used to find the interesting patterns in binary matrices. Several algorithms were designed for computing patterns in binary matrices, but their computational complexity doesn't fit for large datasets. This Paper addresses a proficient concept analysis algorithm for finding neighbors and constructing a concept lattice using MapReduce framework for a given dense formal contexts of high dimensionality in terms of number of objects by modifying Lindig's upper neighbor algorithm for generating concept lattice. The new MapReduce algorithm namely UNConceptGeneration algorithm is implemented using map and Reduce processes to generate formal concepts to construct the concept lattice. The Map process distributes the dense context across various map tasks and computes the concepts, reduce process aggregates all the generated concepts to produce the final outcome. This paper also addresses how the network delays and data transfer rate overcome during the concept lattice generation, which leads to faster concept analysis. The paper describes hadoop's fault tolerance which is one of the key features to choose MapReduce to implement this algorithm in a distributed *** experimental results demonstrate that the proposed approach performs comparable or better than any other parallel algorithms that were tested for sufficiently large datasets. A detailed comparison with other existing parallel and traditional algorithms is also depicted.

关键词： hadoop MapReduce hadoop distributed file system Combiner Formal Concept Analysis

来源：评论

学校读者我要写书评

暂无评论

Chicago Crime Data Analysis Using PIG in hadoop

Chicago Crime Data Analysis Using PIG in Hadoop

引用

IEEE International Conference on Research in Computational Intelligence and Communication Networks

作者： Anupam Mukherjee Sourav De Siddhartha Bhattacharyya Jan Platos Department of Computer Science & Engineering Siliguri Institute of Technology Department of Computer Science & Engineering Cooch Behar Government Engineering College Faculty of Electrical Engineering and Computer Science VSB Technical University of Ostrava

ISBN: (纸本)9781538676394;9781538676387

The need for analyzing the different crime dataset has significantly increased over the past few decades. It is necessary to detect the wide variety of crimes and the corresponding place of occurrences accurately. Government bodies throughout the world maintained Open Data initiatives, which is a large collection of heterogeneous dataset. This enables the government agencies to maintain the law and order of the society. The prime objective of this paper is to analyze the Chicago crime dataset to extract the significant crime information over the years. The proposed analysis is performed to bring out the crime information depending on some predetermined criteria, e.g. total crimes of different types, narcotic crime cases, an offense involving children, analysis of hourly theft cases and location identification where the major theft cases have occurred. The proposed analysis is performed in fully distributed hadoop cluster. Moreover, we have extracted crucial minuscule statistics on theft-related criminal activities and their latitude or longitude.

关键词： Fully distributed hadoop cluster hadoop distributed file system Pig latine Crime data Open data initiatives Name node Data nodes

来源：评论

学校读者我要写书评

暂无评论

Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce

引用

INTERNATIONAL JOURNAL OF DATA WAREHOUSING AND MINING 2016年第1期12卷 49-68页

作者： Esposito, Christian Ficco, Massimo Univ Naples Federico II Naples Italy

The demand to access to a large volume of data, distributed across hundreds or thousands of machines, has opened new opportunities in commerce, science, and computing applications. MapReduce is a paradigm that offers a programming model and an associated implementation for processing massive datasets in a parallel fashion, by using non-dedicated distributed computing hardware. It has been successfully adopted in several academic and industrial projects for Big Data Analytics. However, since such analytics is increasingly demanded within the context of mission-critical applications, security and reliability in MapReduce frameworks are strongly required in order to manage sensible information, and to obtain the right answer at the right time. In this paper, the authors present the main implementation of the MapReduce programming paradigm, provided by Apache with the name of hadoop. They illustrate the security and reliability concerns in the context of a large-scale data processing infrastructure. They review the available solutions, and their limitations to support security and reliability within the context MapReduce frameworks. The authors conclude by describing the undergoing evolution of such solutions, and the possible issues for improvements, which could be challenging research opportunities for academic researchers.

关键词： hadoop distributed file system MapReduce Parallel Processing Reliability Security

来源：评论

学校读者我要写书评

暂无评论

Comparing HiveQL and MapReduce Methods to Process Fact Data in a Data Warehouse 2

Comparing HiveQL and MapReduce Methods to Process Fact Data ...

引用

2nd International Conference on Communication systems, Computing and IT Applications (CSCITA)

作者： Pen, Haince Denis Dsilva, Prajyoti Mascarnes, Sweedle Morgan Stanley New York NY 10036 USA St Francis Inst Technol Dept Informat Technol Mumbai Maharashtra India

ISBN: (纸本)9781509043811

Today Big data is one of the most widely spoken about technology that is being explored throughout the world by technology enthusiasts and academic researchers. The reason for this is the enormous data generated every second of each day. Every webpage visited, every text message sent, every post on social networking websites, check-in information, mouse clicks etc. is logged. This data needs to be stored and retrieved efficiently, moreover the data is unstructured therefore the traditional methods of strong data fail. This data needs to be stored and retrieved efficiently There is a need of an efficient, scalable and robust architecture that needs stores enormous amounts of unstructured data, which can be queried as and when required. In this paper, we come up with a novel methodology to build a data warehouse over big data technologies while specifically addressing the issues of scalability and user performance. Our emphasis is on building a data pipeline which can be used as a reference for future research on the methodologies to build a data warehouse over big data technologies for either structured or unstructured data sources. We have demonstrated the processing of data for retrieving the facts from data warehouse using two techniques, namely HiveQL and MapReduce.

关键词： data warehouse data marts facts dimensions big data hadoop distributed file system MapReduce HiveQL Apache Sqoop

来源：评论

学校读者我要写书评

暂无评论

HDFS Memory Usage Analysis

HDFS Memory Usage Analysis

引用

International Conference on Inventive Computing and Informatics

作者： B. PURNACHANDRA RAO N. NAGAMALLESWARA RAO Dept of Computer Science & Engg ANU College of Engg & Technology Dept of Information Technology R.V.R. & J.C. College of Engg& Technology

ISBN: (纸本)9781538640326

Due to high use of online activities, there will be lot of data which is getting generated. To handle this data there should be an efficient system which can process the data effectively. One such system is hadoop distributed file system. HDFS consists of number of nodes, and one is the master among them while all others are slave nodes. Master node is called as NameNode and slave nodes are called as DataNodes. system will record the activities in the log file. Whatever the transactions are happening at the front end everything will get recorded at backend. We can capture this data and analyze to find out the system behavior in the failure situation. Since we are using the system for large transactions we need to have an idea on usage of the resources in the HDFS such as memory usage. Analyzers can get the stats of the operating system and the other devices separately i.e, for each service we need to have one analyzer to collect the stats. This paper presents the analyzer or shipper for collecting the metrics from operating system, memory and the other installed services.

关键词： hadoop distributed file system HDFS NameNode DataNode ElasticSerach LogStash Kibana MetricBeat Analyzer Shipper

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：