检索结果-内蒙古大学图书馆

5th International Conference on Information Science and Applications (ICISA)

作者： Kong, Yun Hee Park, Young B. Baekseok Univ Dept Informat & Commun Cheonan Si Chungcheongnam South Korea Dankook Univ Dept Comp Sci Yongin 330714 Gyeonggi Do South Korea

ISBN: (纸本)9781479944415

According to data volumes in scientific applications have grown exponentially, new scientific methods to analyze and organize the data are required. mapreduce programming is driving Internet services and those services operation in a cloud environment. Hence it is required to efficiently provide resources for handling diverse mapreduce applications. In this paper we show the Hadoop application with map and reduce functions for the data transformation

关键词： mapreduce programming model hadoop performance evaluation

来源：评论

学校读者我要写书评

暂无评论

BigDataSDNSim: A simulator for analyzing big data applications in software-defined cloud data centers

引用

SOFTWARE-PRACTICE & EXPERIENCE 2021年第5期51卷 893-920页

作者： Alwasel, Khaled Calheiros, Rodrigo N. Garg, Saurabh Buyya, Rajkumar Pathan, Mukaddim Georgakopoulos, Dimitrios Ranjan, Rajiv Newcastle Univ Sch Comp Newcastle Upon Tyne Tyne & Wear England Saudi Elect Univ Coll Comp & Informat Riyadh Saudi Arabia Western Sydney Univ Sch Comp Data & Math Sci Sydney NSW Australia Univ Tasmania Sch Comp & Informat Syst Hobart Tas Australia Univ Melbourne Sch Comp & Informat Syst Melbourne Vic Australia Telstra Corp Ltd Melbourne Vic Australia Swinburne Univ Technol Sch Software & Elect Engn Melbourne Vic Australia

The integration and crosscoordination of big data processing and software-defined networking (SDN) are vital for improving the performance of big data applications. Various approaches for combining big data and SDN have been investigated by both industry and academia. However, empirical evaluations of solutions that combine big data processing and SDN are extremely costly and complicated. To address the problem of effective evaluation of solutions that combine big data processing with SDN, we present a new, self-contained simulation tool named BigDataSDNSim that enables the modeling and simulation of the big data management system YARN, its related programming models mapreduce, and SDN-enabled networks in a cloud computing environment. BigDataSDNSim supports cost-effective and easy to conduct experimentation in a controllable, repeatable, and configurable manner. The article illustrates the simulation accuracy and correctness of BigDataSDNSim by comparing the behavior and results of a real environment that combines big data processing and SDN with an equivalent simulated environment. Finally, the article presents two uses cases of BigDataSDNSim, which exhibit its practicality and features, illustrate the impact of data replication mechanisms of mapreduce in Hadoop YARN, and show the superiority of SDN over traditional networks to improve the performance of mapreduce applications.

关键词： big data joint‐ optimization mapreduce programming model modeling and simulation performance optimization software‐ defined networking

来源：评论

学校读者我要写书评

暂无评论

Performance Improvement of mapreduce for Heterogeneous Clusters Based on Efficient Locality and Replica Aware Scheduling (ELRAS) Strategy

引用

WIRELESS PERSONAL COMMUNICATIONS 2017年第3期95卷 2709-2733页

作者： Benifa, J. V. Bibal Dejey Anna Univ Dept Comp Sci & Engn Reg Campus Tirunelveli 627007 Tamil Nadu India

mapreduce is a parallel programming model for processing the data-intensive applications in a cloud environment. The scheduler greatly influences the performance of mapreduce model while utilized in heterogeneous cluster environment. The dynamic nature of cluster environment and computing workloads affect the execution time and computational resource usage in the scheduling process. Further, data locality is essential for reducing total job execution time, cross-rack communication, and to improve the throughput. In the present work, a scheduling strategy named efficient locality and replica aware scheduling (ELRAS) integrated with an autonomous replication scheme (ARS) is proposed to enhance the data locality and performs consistently in the heterogeneous environment. ARS autonomously decides the data object to be replicated by considering its popularity and removes the replica as it is idle. The proposed approach is validated in a heterogeneous cluster environment with various realistic applications that are IO bound, CPU bound and mixed workloads. ELRAS improves the throughput by a factor about 2 as compared with the existing FIFO and it also yields near optimal data locality, reduce the execution time, and effective utilization of resources. The simplicity of ELRAS algorithm proves its feasibility to adopt for a wide range of applications.

关键词： mapreduce programming model Data locality Heterogeneous clusters Virtualization

来源：评论

学校读者我要写书评

暂无评论

Extraction of mapreduce-based features from spectrograms for audio-based surveillance

引用

DIGITAL SIGNAL PROCESSING 2019年 87卷 1-9页

作者： Mulimani, Manjunath Koolagudi, Shashidhar G. Natl Inst Technol Karnataka Dept Comp Sci & Engn Surathkal 575025 India

In this paper, we proposed a novel parallel method for extraction of significant information from spectrograms using mapreduce programming model for the audio-based surveillance system, which effectively recognizes critical acoustic events in the surrounding environment. Extraction of reliable information as features from spectrograms of big noisy audio event dataset demands high computational time. Parallelizing the feature extraction using mapreduce programming model on Hadoop improves the efficiency of the overall system. The acoustic events with real-time background noise from Mivia lab audio event data set are used for surveillance applications. The proposed approach is time efficient and achieves high performance of recognizing critical acoustic events with the average recognition rate of 96.5% in different noisy conditions. (C) 2019 Elsevier Inc. All rights reserved.

关键词： Acoustic Event Classification (AEC) Audio-based surveillance mapreduce programming model Hadoop

来源：评论

学校读者我要写书评

暂无评论

FiDoop-DP: Data Partitioning in Frequent Itemset Mining on Hadoop Clusters

引用

IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 2017年第1期28卷 101-114页

作者： Xun, Yaling Zhang, Jifu Qin, Xiao Zhao, Xujun Taiyuan Univ Sci & Technol Taiyuan 030024 Shanxi Peoples R China Auburn Univ Dept Comp Sci & Software Engn Samuel Ginn Coll Engn Auburn AL 36849 USA

Traditional parallel algorithms for mining frequent itemsets aim to balance load by equally partitioning data among a group of computing nodes. We start this study by discovering a serious performance problem of the existing parallel Frequent Itemset Mining algorithms. Given a large dataset, data partitioning strategies in the existing solutions suffer high communication and mining overhead induced by redundant transactions transmitted among computing nodes. We address this problem by developing a data partitioning approach called FiDoop-DP using the mapreduce programming model. The overarching goal of FiDoop-DP is to boost the performance of parallel Frequent Itemset Mining on Hadoop clusters. At the heart of FiDoop-DP is the Voronoi diagram-based data partitioning technique, which exploits correlations among transactions. Incorporating the similarity metric and the Locality-Sensitive Hashing technique, FiDoop-DP places highly similar transactions into a data partition to improve locality without creating an excessive number of redundant transactions. We implement FiDoop-DP on a 24-node Hadoop cluster, driven by a wide range of datasets created by IBM Quest Market-Basket Synthetic Data Generator. Experimental results reveal that FiDoop-DP is conducive to reducing network and computing loads by the virtue of eliminating redundant transactions on Hadoop nodes. FiDoop-DP significantly improves the performance of the existing parallel frequent-pattern scheme by up to 31 percent with an average of 18 percent.

关键词： Frequent itemset mining parallel data mining data partitioning mapreduce programming model hadoop cluster

来源：评论

学校读者我要写书评

暂无评论

Optimal Capacity Allocation for executing mapreduce Jobs in Cloud Systems 16

Optimal Capacity Allocation for executing MapReduce Jobs in ...

引用

16th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC)

作者： Malekimajd, M. Rizzi, A. M. Ardagna, D. Ciavotta, M. Passacantando, M. Movaghar, A. Sharif Univ Technol Dept Comp Engn Tehran Iran Politecn Milan Dipartimento Elettron Informaz & Bioingn I-20133 Milan Italy Univ Pisa Dipartimento Informat Pisa Italy

ISBN: (纸本)9781479984480

Nowadays, analyzing large amount of data is of paramount importance for many companies. Big data and business intelligence applications are facilitated by the mapreduce programming model while, at infrastructural layer, cloud computing provides flexible and cost effective solutions for allocating on demand large clusters. Capacity allocation in such systems is a key challenge to providing performance for mapreduce jobs and minimize cloud resource cost. The contribution of this paper is twofold: (i) we formulate a linear programming model able to minimize cloud resources cost and job rejection penalties for the execution of jobs of multiple classes with (soft) deadline guarantees, (ii) we provide new upper and lower bounds for mapreduce job execution time in shared Hadoop clusters. Moreover, our solutions are validated by a large set of experiments. We demonstrate that our method is able to determine the global optimal solution for systems including up to 1000 user classes in less than 0.5 seconds. Moreover, the execution time of mapreduce jobs are within 19% of our upper bounds on average.

关键词： Big Data cloud computing data analysis linear programming parallel programming pattern clustering resource allocation mapreduce job execution time mapreduce programming model big data applications business intelligence applications capacity allocation cloud resource cost minimization cloud systems infrastructural layer job rejection linear programming model optimal capacity allocation shared Hadoop clusters Bismuth Mathematical model Optimization Resource management Scalability Silicon Upper bound Capacity Allocation Cloud Computing mapreduce Performance bounds Cloud Computing big data linear programming models management of resources Upper bound jobs Parallel programming Data analysis Scalability bismuth Cluster Analysis linear programming Resource utilization Mathematical model Big data applications

来源：评论

学校读者我要写书评

暂无评论

MapFIM: Memory Aware Parallelized Frequent Itemset Mining in Very Large Datasets 1

引用

28th International Conference on Database and Expert Systems Applications (DEXA)

作者： Duong, Khanh-Chuong Bamha, Mostafa Giacometti, Arnaud Li, Dominique Soulet, Arnaud Vrain, Christel Univ Francois Rabelais Tours LI EA 6300 Blois France Univ Orleans INSA Ctr Val de Loire LIFO EA 4022 Blois France

ISBN: (数字)9783319644684

ISBN: (纸本)9783319644684;9783319644677

Mining frequent itemsets in large datasets has received much attention, in recent years, relying on mapreduce programming models. Many famous FIM algorithms have been parallelized in a mapreduce framework like Parallel Apriori, Parallel FP-Growth and Dist-Eclat. However, most papers focus on work partitioning and/or load balancing but they are not extensible because they require some memory assumptions. A challenge in designing parallel FIM algorithms is thus finding ways to guarantee that data structures used during mining always fit in the local memory of the processing nodes during all computation steps. In this paper, we propose MapFIM, a two-phase approach for frequent itemset mining in very large datasets relying both on a mapreduce-based distributed Apriori method and a local in-memory method. In our approach, mapreduce is first used to generate local memory-fitted prefix-projected databases from the input dataset benefiting from the Apriori principle. Then an optimized local in-memory mining process is launched to generate all frequent itemsets from each prefix-projected database. Performance evaluation shows that MapFIM is more efficient and more extensible than existing mapreduce based frequent itemset mining approaches.

关键词： Frequent itemset mining mapreduce programming model Distributed file systems Hadoop framework

来源：评论

学校读者我要写书评

暂无评论

Measuring Documents Similarity in Large Corpus using mapreduce Algorithm 5

Measuring Documents Similarity in Large Corpus using MapRedu...

引用

5th International Conference on Multimedia Computing and Systems (ICMCS)

作者： Birjali, Marouane Beni-Hssane, Abderrahim Erritali, Mohammed Univ Chouaib Doukkali Dept Comp Sci Fac Sci El Jadida Morocco Univ Sultan Moulay Slimane Dept Comp Sci Fac Sci & Technol Beni Mellal Morocco

ISBN: (纸本)9781509051465

Document similarity measures between documents and queries has been extensively studied in information retrieval. Measuring the similarity of documents are crucial components of many text-analysis tasks, including information retrieval, document classification, and document clustering. However, there are a growing number of tasks that require computing the similarity between two very short segments of text. There exist a large number of composed documents in a large amount of corpus. Most of them are required to compute the similarity for validation. In this paper, we propose our approach of measuring similarity between documents in large amount of corpus. For evaluation, we compare the proposed approach with other approaches previously presented by using our new mapreduce algorithm. Simulation results, on Hadoop framework, show that our new mapreduce algorithm outperforms the classical ones in term of running time performance and increases the value of the similarity.

关键词： Hadoop cluster document similarity mapreduce programming model similarity measure

来源：评论

学校读者我要写书评

暂无评论

Parallel Information Fusion Method for Microarray Data Analysis 3

Parallel Information Fusion Method for Microarray Data Analy...

引用

IEEE International Conference on Big Data

作者： Meng, Jun Li, Rui Zhang, Jing Dalian Univ Technol Sch Comp Sci & Technol Dalian Peoples R China

ISBN: (纸本)9781479999255

Classification of microarray data has always been a challenging task due to the enormous number of genes. Finding a small, closely related gene set to accurately classify disease cells is an important research problem. Integrating biological knowledge into genomic analysis to help to improve the interpretation of the results is an effective approach. In this paper, affinity propagation (AP) clustering algorithm is chosen to analyze the impact of the biological similarity on the results. We integrate GO semantic similarity into AP clustering for granule construction. Using mapreduce programming model, a parallel information fusion method is proposed. The process of similarity matrix construction and message passing in AP algorithm is parallelized using mapreduce. Parallel randomly directed hill climb ensemble pruning (RandomDHCEP) method based on mapreduce is introduced for ensemble pruning. An instance analysis represents the process of affinity propagation and ensemble pruning by using iterative mapreduce program. The proposed method can offer good scalability on large data with increasing number of nodes and it can also provide higher classification accuracy rather than using whole gene set for classification.

关键词： Microarray Data mapreduce programming model Parallel Information Fusion

来源：评论

学校读者我要写书评

暂无评论

Towards Scalability and Data Skew Handling in GroupBy-Joins using mapreduce model

Towards Scalability and Data Skew Handling in GroupBy-Joins ...

引用

15th Annual International Conference on Computational Science (ICCS)

作者： Hassan, M. Al Hajj Bamha, M. Lebanese Int Univ Beirut Lebanon Univ Orleans INSA Ctr Val Loire F-45067 Orleans France

For over a decade, mapreduce has become the leading programming model for parallel and massive processing of large volumes of data. This has been driven by the development of many frameworks such as Spark, Pig and Hive, facilitating data analysis on large-scale systems. However, these frameworks still remain vulnerable to communication costs, data skew and tasks imbalance problems. This can have a devastating effect on the performance and on the scalability of these systems, more particularly when treating GroupBy-Join queries of large datasets. In this paper, we present a new GroupBy-Join algorithm allowing to reduce communication costs considerably while avoiding data skew effects. A cost analysis of this algorithm shows that our approach is insensitive to data skew and ensures perfect balancing properties during all stages of GroupBy-Join computation even for highly skewed data. These performances have been confirmed by a series of experimentations.

关键词： Join and GrouBy-join operations Data skew mapreduce programming model Distributed file systems Hadoop framework Apache Pig Latin

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：