检索结果-内蒙古大学图书馆

5th International Conference on Information Science and Applications (ICISA)

作者： Kong, Yun Hee Park, Young B. Baekseok Univ Dept Informat & Commun Cheonan Si Chungcheongnam South Korea Dankook Univ Dept Comp Sci Yongin 330714 Gyeonggi Do South Korea

ISBN: (纸本)9781479944415

According to data volumes in scientific applications have grown exponentially, new scientific methods to analyze and organize the data are required. mapreduce programming is driving Internet services and those services operation in a cloud environment. Hence it is required to efficiently provide resources for handling diverse mapreduce applications. In this paper we show the Hadoop application with map and reduce functions for the data transformation

关键词： mapreduce programming model hadoop performance evaluation

来源：评论

学校读者我要写书评

暂无评论

Hadoop-based framework for big data analysis of synchronised harmonics in active distribution network

引用

IET GENERATION TRANSMISSION & DISTRIBUTION 2017年第16期11卷 3930-3937页

作者： Cao, Zijian Lin, Jin Wan, Can Song, Yonghua Taylor, Gareth Li, Maozhen Tsinghua Univ Dept Elect Engn Beijing Peoples R China Zhejiang Univ Coll Elect Engn Hangzhou Zhejiang Peoples R China Brunel Univ Brunel Inst Power Syst London England Brunel Univ Sch Engn & Design London England

Synchronised harmonic analysis that utilises GPS signals to synchronise harmonic measurements enables advanced power quality analysis in smart grid. However, it is difficult to put this promising technology into practice due to an extremely high requirement on data storage and processing. To overcome this problem, this study presents a novel Hadoop-based framework for big data analysis of synchronised harmonics in distribution networks. The proposed framework facilitates harmonic data processing associated with data storage and advanced analysis based on big data techniques. Under this framework, a big matrix multiplication algorithm based on mapreduce programming model solving the harmonic state estimation problem is deployed in the stage of data processing. A mapreduce-based harmonics distortion calculation is implemented as an advanced analysis for harmonics. Comprehensive numerical studies are carried out to analyse the characteristics of the proposed algorithm and verify the effectiveness of the proposed framework.

关键词： Global Positioning System smart grid GPS signals advanced power quality analysis big data analysis mapreduce-based harmonic distortion calculation power distribution power supply quality Hadoop-based framework harmonic distortion Radionavigation and direction finding data analysis Data handling techniques synchronisation active distribution network harmonic data processing Power supply quality and harmonics big matrix multiplication algorithm synchronised harmonic analysis Distribution networks big data technique Big Data mapreduce programming model data storage power engineering computing harmonic measurements harmonic state estimation problem smart power grids Power engineering computing

来源：评论

学校读者我要写书评

暂无评论

Design of Wireless Sensor Network Data Acquisition System via Health Sensor Based on Symmetric Encryption Algorithm

引用

JOURNAL OF TESTING AND EVALUATION 2023年第1期51卷 278-290页

作者： Xuan, Chunqing Zhengzhou Business Univ Coll Informat & Elect Engn Dept Comp Sci & Technol 136 Zijing Rd Zhengzhou 451200 Peoples R China

In order to improve the data collection effect of the wireless sensor network, a data collection system based on symmetric encryption algorithm is designed via health sensor. Upload the received data to the host via RS-232 to get the working mode and clock activity. The data acquisition circuit is designed with MSP430 module. The mapreduce programming model is used to complete data collection, a symmetric encryption algorithm is introduced, and a range data encryption query scheme with privacy protection function is designed. Apply it to the node data of the wireless sensor network to realize the secure data collection of the wireless sensor network. Experimental results show that the system has the advantages of high efficiency, large amount of data collection, and high residual energy of sensor network nodes.

关键词： symmetric encryption algorithm wireless sensor network data acquisition MSP430 analog-to-digital conversion module mapreduce programming model kinematic algorithm

来源：评论

学校读者我要写书评

暂无评论

Performance Improvement of mapreduce for Heterogeneous Clusters Based on Efficient Locality and Replica Aware Scheduling (ELRAS) Strategy

引用

WIRELESS PERSONAL COMMUNICATIONS 2017年第3期95卷 2709-2733页

作者： Benifa, J. V. Bibal Dejey Anna Univ Dept Comp Sci & Engn Reg Campus Tirunelveli 627007 Tamil Nadu India

mapreduce is a parallel programming model for processing the data-intensive applications in a cloud environment. The scheduler greatly influences the performance of mapreduce model while utilized in heterogeneous cluster environment. The dynamic nature of cluster environment and computing workloads affect the execution time and computational resource usage in the scheduling process. Further, data locality is essential for reducing total job execution time, cross-rack communication, and to improve the throughput. In the present work, a scheduling strategy named efficient locality and replica aware scheduling (ELRAS) integrated with an autonomous replication scheme (ARS) is proposed to enhance the data locality and performs consistently in the heterogeneous environment. ARS autonomously decides the data object to be replicated by considering its popularity and removes the replica as it is idle. The proposed approach is validated in a heterogeneous cluster environment with various realistic applications that are IO bound, CPU bound and mixed workloads. ELRAS improves the throughput by a factor about 2 as compared with the existing FIFO and it also yields near optimal data locality, reduce the execution time, and effective utilization of resources. The simplicity of ELRAS algorithm proves its feasibility to adopt for a wide range of applications.

关键词： mapreduce programming model Data locality Heterogeneous clusters Virtualization

来源：评论

学校读者我要写书评

暂无评论

Extraction of mapreduce-based features from spectrograms for audio-based surveillance

引用

DIGITAL SIGNAL PROCESSING 2019年 87卷 1-9页

作者： Mulimani, Manjunath Koolagudi, Shashidhar G. Natl Inst Technol Karnataka Dept Comp Sci & Engn Surathkal 575025 India

In this paper, we proposed a novel parallel method for extraction of significant information from spectrograms using mapreduce programming model for the audio-based surveillance system, which effectively recognizes critical acoustic events in the surrounding environment. Extraction of reliable information as features from spectrograms of big noisy audio event dataset demands high computational time. Parallelizing the feature extraction using mapreduce programming model on Hadoop improves the efficiency of the overall system. The acoustic events with real-time background noise from Mivia lab audio event data set are used for surveillance applications. The proposed approach is time efficient and achieves high performance of recognizing critical acoustic events with the average recognition rate of 96.5% in different noisy conditions. (C) 2019 Elsevier Inc. All rights reserved.

关键词： Acoustic Event Classification (AEC) Audio-based surveillance mapreduce programming model Hadoop

来源：评论

学校读者我要写书评

暂无评论

FiDoop-DP: Data Partitioning in Frequent Itemset Mining on Hadoop Clusters

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2017年第1期28卷 101-114页

作者： Xun, Yaling Zhang, Jifu Qin, Xiao Zhao, Xujun Taiyuan Univ Sci & Technol Taiyuan 030024 Shanxi Peoples R China Auburn Univ Dept Comp Sci & Software Engn Samuel Ginn Coll Engn Auburn AL 36849 USA

Traditional parallel algorithms for mining frequent itemsets aim to balance load by equally partitioning data among a group of computing nodes. We start this study by discovering a serious performance problem of the existing parallel Frequent Itemset Mining algorithms. Given a large dataset, data partitioning strategies in the existing solutions suffer high communication and mining overhead induced by redundant transactions transmitted among computing nodes. We address this problem by developing a data partitioning approach called FiDoop-DP using the mapreduce programming model. The overarching goal of FiDoop-DP is to boost the performance of parallel Frequent Itemset Mining on Hadoop clusters. At the heart of FiDoop-DP is the Voronoi diagram-based data partitioning technique, which exploits correlations among transactions. Incorporating the similarity metric and the Locality-Sensitive Hashing technique, FiDoop-DP places highly similar transactions into a data partition to improve locality without creating an excessive number of redundant transactions. We implement FiDoop-DP on a 24-node Hadoop cluster, driven by a wide range of datasets created by IBM Quest Market-Basket Synthetic Data Generator. Experimental results reveal that FiDoop-DP is conducive to reducing network and computing loads by the virtue of eliminating redundant transactions on Hadoop nodes. FiDoop-DP significantly improves the performance of the existing parallel frequent-pattern scheme by up to 31 percent with an average of 18 percent.

关键词： Frequent itemset mining parallel data mining data partitioning mapreduce programming model hadoop cluster

来源：评论

学校读者我要写书评

暂无评论

MapFIM: Memory Aware Parallelized Frequent Itemset Mining in Very Large Datasets 1

引用

28th International Conference on Database and Expert Systems Applications (DEXA)

作者： Duong, Khanh-Chuong Bamha, Mostafa Giacometti, Arnaud Li, Dominique Soulet, Arnaud Vrain, Christel Univ Francois Rabelais Tours LI EA 6300 Blois France Univ Orleans INSA Ctr Val de Loire LIFO EA 4022 Blois France

ISBN: (数字)9783319644684

ISBN: (纸本)9783319644684;9783319644677

Mining frequent itemsets in large datasets has received much attention, in recent years, relying on mapreduce programming models. Many famous FIM algorithms have been parallelized in a mapreduce framework like Parallel Apriori, Parallel FP-Growth and Dist-Eclat. However, most papers focus on work partitioning and/or load balancing but they are not extensible because they require some memory assumptions. A challenge in designing parallel FIM algorithms is thus finding ways to guarantee that data structures used during mining always fit in the local memory of the processing nodes during all computation steps. In this paper, we propose MapFIM, a two-phase approach for frequent itemset mining in very large datasets relying both on a mapreduce-based distributed Apriori method and a local in-memory method. In our approach, mapreduce is first used to generate local memory-fitted prefix-projected databases from the input dataset benefiting from the Apriori principle. Then an optimized local in-memory mining process is launched to generate all frequent itemsets from each prefix-projected database. Performance evaluation shows that MapFIM is more efficient and more extensible than existing mapreduce based frequent itemset mining approaches.

关键词： Frequent itemset mining mapreduce programming model Distributed file systems Hadoop framework

来源：评论

学校读者我要写书评

暂无评论

Parallel Information Fusion Method for Microarray Data Analysis 3

Parallel Information Fusion Method for Microarray Data Analy...

引用

IEEE International Conference on Big Data

作者： Meng, Jun Li, Rui Zhang, Jing Dalian Univ Technol Sch Comp Sci & Technol Dalian Peoples R China

ISBN: (纸本)9781479999255

Classification of microarray data has always been a challenging task due to the enormous number of genes. Finding a small, closely related gene set to accurately classify disease cells is an important research problem. Integrating biological knowledge into genomic analysis to help to improve the interpretation of the results is an effective approach. In this paper, affinity propagation (AP) clustering algorithm is chosen to analyze the impact of the biological similarity on the results. We integrate GO semantic similarity into AP clustering for granule construction. Using mapreduce programming model, a parallel information fusion method is proposed. The process of similarity matrix construction and message passing in AP algorithm is parallelized using mapreduce. Parallel randomly directed hill climb ensemble pruning (RandomDHCEP) method based on mapreduce is introduced for ensemble pruning. An instance analysis represents the process of affinity propagation and ensemble pruning by using iterative mapreduce program. The proposed method can offer good scalability on large data with increasing number of nodes and it can also provide higher classification accuracy rather than using whole gene set for classification.

关键词： Microarray Data mapreduce programming model Parallel Information Fusion

来源：评论

学校读者我要写书评

暂无评论

Towards Scalability and Data Skew Handling in GroupBy-Joins using mapreduce model

Towards Scalability and Data Skew Handling in GroupBy-Joins ...

引用

15th Annual International Conference on Computational Science (ICCS)

作者： Hassan, M. Al Hajj Bamha, M. Lebanese Int Univ Beirut Lebanon Univ Orleans INSA Ctr Val Loire F-45067 Orleans France

For over a decade, mapreduce has become the leading programming model for parallel and massive processing of large volumes of data. This has been driven by the development of many frameworks such as Spark, Pig and Hive, facilitating data analysis on large-scale systems. However, these frameworks still remain vulnerable to communication costs, data skew and tasks imbalance problems. This can have a devastating effect on the performance and on the scalability of these systems, more particularly when treating GroupBy-Join queries of large datasets. In this paper, we present a new GroupBy-Join algorithm allowing to reduce communication costs considerably while avoiding data skew effects. A cost analysis of this algorithm shows that our approach is insensitive to data skew and ensures perfect balancing properties during all stages of GroupBy-Join computation even for highly skewed data. These performances have been confirmed by a series of experimentations.

关键词： Join and GrouBy-join operations Data skew mapreduce programming model Distributed file systems Hadoop framework Apache Pig Latin

来源：评论

学校读者我要写书评

暂无评论

Affinity-aware Virtual Cluster Optimization for mapreduce Applications

Affinity-aware Virtual Cluster Optimization for MapReduce Ap...

引用

IEEE International Conference on Cluster Computing

作者： Yan, Cairong Zhu, Ming Yang, Xin Yu, Ze Li, Min Shi, Youqun Li, Xiaolin Donghua Univ Sch Comp Sci & Technol Shanghai Peoples R China Univ Florida Scalable Software Syst Lab Gainesville FL 32611 USA

ISBN: (纸本)9781467324229

Infrastructure-as-a-Service clouds are becoming ubiquitous for provisioning virtual machines on demand. Cloud service providers expect to use least resources to deliver best services. As users frequently request virtual machines to build virtual clusters and run mapreduce-like jobs for big data processing, cloud service providers intend to place virtual machines closely to minimize network latency and subsequently reduce data movement cost. In this paper we focus on the virtual machine placement issue for provisioning virtual clusters with minimum network latency in clouds. We define distance as the latency between virtual machines and use it to measure the affinity of virtual clusters. Such metric of distance indicates the considerations of virtual machine placement and topology of physical nodes in clouds. Then we formulate our problem as the classical shortest distance problem and solve it by modeling to integer programming problem. A greedy virtual machine placement algorithm is designed to get a compact virtual cluster. Furthermore, an improved heuristic algorithm is also presented for achieving a global resource optimization. The simulation results verify our algorithms and the experiment results validate the improvement achieved by our approaches.

关键词： Virtual cluster mapreduce programming model Provisioning Shortest distance Resource optimization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：