检索结果-内蒙古大学图书馆

DupM: a Data Replica Allocation Strategy for distributed mining

Procedia Computer Science 2021年 188卷 137-143页

作者： Maoran Wang College of Computer and Information Science College of Software Southwest University Chongqing and 400715 China

The mass data mining algorithm is usually sensitive to computing resources, the quantities of which (e.g., the quantities of memory and CPUs available for mining algorithms) determine the mining results. In this paper, data replicas as a resource were incorporated into the resource scheduling strategy for distributed data mining to offer a replica-aware resource scheduling strategy DupM. With data replicas as a type of data resources, the DupM strategy scheduled data replicas through dynamic programming so that a cloud platform could, based on the requirements of distributed mining, allocate replicas. The simulation test of test data sets in KDD CUP events and IBM synthesizer data sets demonstrated that the DupM resource scheduling strategy has more advantages than Hadoop’s built-in resource scheduling strategy.

关键词： replica scheduling distributed mining dynamic programming Hadoop

来源：评论

学校读者我要写书评

暂无评论

A parallel and distributed C4.5 algorithm in cloud computing environments

引用

COMPUTING 2025年第2期107卷 1-38页

作者： Lin, Kawuu W. Zheng, Ya-Jun Chen, Ju-Chin Wang, Wei-Chiang Chen, Chao-Chun Natl Kaohsiung Univ Sci & Technol Dept Comp Sci & Informat Engn Jiangong Rd Kaohsiung 807618 Taiwan Natl Cheng Kung Univ Inst Mfg Informat & Syst Univ Rd Tainan 70101 Taiwan

Data mining seeks to derive significant insights from big data, offering valuable information for decision-makers and generating economic value. Among its tasks, classification is one of the most essential, with decision tree algorithms being a widely adopted solution due to their efficiency in addressing association rules and classification challenges. Decision trees are particularly advantageous for their ability to provide interpretable results at minimal computational cost. However, the exponential growth of data in the Internet era has highlighted the limitations of traditional algorithms, which struggle to efficiently process large-scale *** address this issue, this study introduces PD-C4.5, a parallel and distributed implementation of the C4.5 algorithm designed for cloud computing environments. By incorporating a microservices architecture, the proposed approach modularizes computation, enabling flexible, scalable, and distributed execution. This design not only enhances system maintainability but also optimizes resource utilization while maintaining high computational *** evaluations demonstrate that PD-C4.5 significantly outperforms Original C4.5 and MR-C4.5 in handling medium and large datasets, achieving notable reductions in computation time and resource consumption. Additionally, the integration of microservices ensures seamless scalability to accommodate increasing data volumes. This study provides a novel and practical solution for large-scale data classification, bridging the gap between computational efficiency and scalability in the era of big data.

关键词： Data mining Classification algorithm C4.5 distributed mining

来源：评论

学校读者我要写书评

暂无评论

An Efficient Framework for mining Association Rules in the distributed Databases

引用

COMPUTER JOURNAL 2018年第5期61卷 645-657页

作者： Goyal, Lalit Mohan Beg, M. M. Sufyan Ahmad, Tanvir Bharati Vidyapeeths Coll Engn Comp Sci & Engn New Delhi India Aligarh Muslim Univ Fac Engn & Technol Comp Engn Aligarh Uttar Pradesh India Jamia Millia Islamia Comp Engn New Delhi India

While mining the association rules in distributed database, overhead increases at each site because of linkage and dependency with other sites. Each site scans database not only for itself but for the neighboring sites also. In the most popular Count Distribution (CD) and Fast distributed mining (FDM) algorithms, sites generate and scan the identical candidate itemsets. In the CD algorithm, sites generate candidate k + 1 itemsets using global frequent k-itemsets and in the FDM algorithm, sites generate using its own and neighboring sites heavy frequent k-itemsets. Most of the itemsets scanned by the CD algorithm are infrequent. These infrequent itemsets are not scanned in the FDM algorithm. Anyhow, in the FDM algorithm, some of the itemsets may be found frequent on neither of the sites but scanned on all the sites. In this paper, an efficient framework and an algorithm have been proposed for mining association rules in the distributed database. In the proposed framework, initially, overhead of each site for generating and scanning candidate itemsets for the neighboring sites is reduced. Later, a site either does not scan candidate k-itemset of neighboring site or postpone till its k + 1 itemsets are scanned.

关键词： distributed mining distributed algorithms distributed database frequent itemsets

来源：评论

学校读者我要写书评

暂无评论

A fast and low idle time method for mining frequent patterns in distributed and many-task computing environments

引用

distributed AND PARALLEL DATABASES 2018年第4期36卷 613-641页

作者： Lin, Chun-Cheng Chung, Sheng-Hao Chen, Ju-Chin Yu, Yuan-Tse Lin, Kawuu W. Natl Chiao Tung Univ Dept Ind Engn & Management Hsinchu Taiwan Natl Kaohsiung Univ Sci & Technol Dept Comp Sci & Informat Engn Kaohsiung Taiwan Natl Kaohsiung Normal Univ Dept Software Engn & Management Kaohsiung Taiwan

Association rules mining has attracted much attention among data mining topics because it has been successfully applied in various fields to find the association between purchased items by identifying frequent patterns (FPs). Currently, databases are huge, ranging in size from terabytes to petabytes. Although past studies can effectively discover FPs to deduce association rules, the execution efficiency is still a critical problem, particularly for big data. Progressive size working set (PSWS) and parallel FP-growth (PFP) are state-of-the-art methods that have been applied successfully to parallel and distributed computing technology to improve mining processing time in many-task computing, thereby bridging the gap between high-throughput and high-performance computing. However, such methods cannot mine before obtaining a complete FP-tree or the corresponding subdatabase, causing a high idle time for computing nodes. We propose a method that can begin mining when a small part of an FP-tree is received. The idle time of computing nodes can be reduced, and thus, the time required for mining can be reduced effectively. Through an empirical evaluation, the proposed method is shown to be faster than PSWS and PFP.

关键词： distributed mining distributed computing Frequent pattern mining Many-task computing

来源：评论

学校读者我要写书评

暂无评论

A mining Algorithm for distributed Global Maximal Frequent Itemsets based on Sorted SCan-Tree

A Mining Algorithm for Distributed Global Maximal Frequent I...

引用

2016 IEEE Advanced Information Management,Communicates,Electronic and Automation Control Conference(IMCEC 2016)

作者： Yulei Huang Jinhuan Wang Yan Li Qing Lin ZTE Telecommunication College Xi'an Peihua Universty

ISBN: (纸本)9781509001668

A new algorithm,named SCan-MAX for mining distributed maximal frequent itemsets from databases was proposed,the SCan-MAX used Sorted SCan-tree to store all the information of the transactions from the ***-MAX firstly scanned the local database and then gets the global 1-frequent itemsets,and then it created the SCan-tree on each node and used orderly sequence to store frequent *** the local SCan-tree was send to the master node to create the global ***,the root's sub-tree was send to the slave node to mine local maximal frequent itemsets and send the information of itemsets to the master node to mine the global maximal frequent *** results show that the algorithm makes full use of the characteristics of SCan-Tree,significantly reduces the amount of communication required between nodes,thus enhanced the efficiency of the *** analysis and experimental results show that the algorithm has a good time and space efficiency.

关键词： Data mining Association rule distributed mining Maximal Frequent Itemsets SCan-Tree

来源：评论

学校读者我要写书评

暂无评论

A fast and resource efficient mining algorithm for discovering frequent patterns in distributed computing environments

引用

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2015年 52卷 49-58页

作者： Lin, Kawuu W. Chung, Sheng-Hao Natl Kaohsiung Univ Appl Sci Dept Comp Sci & Informat Engn Kaohsiung 807 Taiwan

The advancement of electronic technology enables us to collect logs from various devices. Such logs require detailed analysis in order to be broadly useful. Data mining is a technique that has been widely used to extract hidden information from such data. Data mining is mainly composed of association rules mining, sequent pattern mining, classification and clustering. Association rules mining has attracted significant attention and been successfully applied to various fields. Although the past studies can effectively discover frequent patterns to deduce association rules, execution efficiency is still a critical problem. To speed up execution, many methods using parallel and distributed computing technology have been proposed in recent years. Most of the past studies focused on parallelizing the workload in a high end machine or in distributed computing environments like grid or cloud computing systems;however, very few of them discuss how to efficiently determine the appropriate number of computing nodes, considering execution efficiency and load balancing. An intuition is that execution speed is proportional to the number of computing nodes that is, more the number of computing nodes, faster is the execution speed. However, this is incorrect for such algorithms because of the inherently algorithmic design. Allocating too many computing nodes can lead to high execution time. In addition to the execution inefficiency, inappropriate resource allocation is a waste of computing power and network bandwidth. At the same time, load cannot be effectively distributed if there are too few nodes allocated. In this paper, we propose a fast, load balancing and resource efficient algorithm named FLR-mining for discovering frequent patterns in distributed computing systems. FLR-mining is capable of determining the appropriate number of computing nodes automatically and achieving better load balancing as compared with existing methods. Through empirical evaluation, FLR-mining is s

关键词： Data mining Frequent pattern mining distributed mining Parallel mining

来源：评论

学校读者我要写书评

暂无评论

Hierarchical Aggregation Approach for distributed clustering of spatial datasets 16

Hierarchical Aggregation Approach for Distributed clustering...

引用

16th IEEE International Conference on Data mining (ICDM)

作者： Bendechache, Malika Le-Khac, Nhien-An Kechadi, M-Tahar Univ Coll Dublin Sch Comp Sci Insight Ctr Data Analyt Dublin Ireland Univ Coll Dublin Sch Comp Sci Dublin Ireland

ISBN: (纸本)9781509059102

In this paper, we present a new approach of distributed clustering for spatial datasets, based on an innovative and efficient aggregation technique. This distributed approach consists of two phases: 1) local clustering phase, where each node performs a clustering on its local data, 2) aggregation phase, where the local clusters are aggregated to produce global clusters. This approach is characterised by the fact that the local clusters are represented in a simple and efficient way. And The aggregation phase is designed in such a way that the final clusters are compact and accurate while the overall process is efficient in both response time and memory allocation. We evaluated the approach with different datasets and compared it to well-known clustering techniques. The experimental results show that our approach is very promising and outperforms all those algorithms.

关键词： Big Data spatial data clustering distributed mining data analysis k-means DBSCAN balance vector

来源：评论

学校读者我要写书评

暂无评论

Determining the appropriate number of nodes for fast mining of frequent patterns in distributed computing environments

引用

INTERNATIONAL JOURNAL OF PARALLEL EMERGENT AND distributed SYSTEMS 2015年第5期30卷 380-392页

作者： Lin, Wei-Tee Chu, Chih-Ping Natl Cheng Kung Univ Dept Comp Sci & Informat Engn Tainan Taiwan

The rapid growth of data brought new challenges, the execution efficiency and scalability, for mining frequent patterns (FPs). To accelerate the execution, many algorithms based on the generate-and-test approach or FP-growth utilising parallel distributed technologies have been proposed in recent years. Most of the past studies focused on designing efficient mining algorithm, and none of them has explored how the appropriate number of computing nodes is determined. Using too many computing nodes will increase the execution time because the existing algorithms need to transmit the sub-databases or FP-trees over the network;using insufficient computing nodes may not effectively distribute the mining loading. In this article, we propose a novel algorithm for efficiently mining FPs with the ability to determine the appropriate number of computing nodes in distributed computing environments. Through empirical evaluations in various simulation conditions, the proposed algorithm is shown to deliver excellent performance in terms of execution time.

关键词： frequent pattern mining parallel mining distributed mining

来源：评论

学校读者我要写书评

暂无评论

distributed Clustering Algorithm for Spatial Data mining 2

Distributed Clustering Algorithm for Spatial Data Mining

引用

2nd IEEE International Conference on Spatial Data mining and Geographical Knowledge Services (ICSDM)

作者： Bendechache, Malika Kechadi, M-Tahar Univ Coll Dublin Sch Comp Sci & Informat Dublin 04 Ireland

ISBN: (纸本)9781479977499

distributed data mining techniques and mainly distributed clustering are widely used in the last decade because they deal with very large and heterogeneous datasets which cannot be gathered centrally. Current distributed clustering approaches are normally generating global models by aggregating local results that are obtained on each site. While this approach mines the datasets on their locations the aggregation phase is complex, which may produce incorrect and ambiguous global clusters and therefore incorrect knowledge. In this paper we propose a new clustering approach for very large spatial datasets that are heterogeneous and distributed. The approach is based on K-means Algorithm but it generates the number of global clusters dynamically. Moreover, this approach uses an elaborated aggregation phase. The aggregation phase is designed in such a way that the overall process is efficient in time and memory allocation. Preliminary results show that the proposed approach produces high quality results and scales up well. We also compared it to two popular clustering algorithms and show that this approach is much more efficient.

关键词： Spatial data Clustering distributed mining Data analysis K-means

来源：评论

学校读者我要写书评

暂无评论

Efficient Large Scale Clustering based on Data Partitioning 3

Efficient Large Scale Clustering based on Data Partitioning

引用

3rd IEEE/ACM International Conference on Data Science and Advanced Analytics (DSAA)

作者： Bendechache, Malika Le-Khac, Nhien-An Kechadi, M-Tahar Univ Coll Dublin Sch Comp Sci Insight Ctr Data Analyt Dublin Ireland Univ Coll Dublin Sch Comp Sci Dublin Ireland

ISBN: (纸本)9781509052066

Clustering techniques are very attractive for extracting and identifying patterns in datasets. However, their application to very large spatial datasets presents numerous challenges such as high-dimensionality data, heterogeneity, and high complexity of some algorithms. For instance, some algorithms may have linear complexity but they require the domain knowledge in order to determine their input parameters. distributed clustering techniques constitute a very good alternative to the big data challenges (e.g., Volume, Variety, Veracity, and Velocity). Usually these techniques consist of two phases. The first phase generates local models or patterns and the second one tends to aggregate the local results to obtain global models. While the first phase can be executed in parallel on each site and, therefore, efficient, the aggregation phase is complex, time consuming and may produce incorrect and ambiguous global clusters and therefore incorrect models. In this paper we propose a new distributed clustering approach to deal efficiently with both phases;generation of local results and generation of global models by aggregation. For the first phase, our approach is capable of analysing the datasets located in each site using different clustering techniques. The aggregation phase is designed in such a way that the final clusters are compact and accurate while the overall process is efficient in time and memory allocation. For the evaluation, we use two well-known clustering algorithms;K-Means and DBSCAN. One of the key outputs of this distributed clustering technique is that the number of global clusters is dynamic;no need to be fixed in advance. Experimental results show that the approach is scalable and produces high quality results.

关键词： Big Data spatial data clustering distributed mining data analysis k-means DBSCAN

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：