With the rise of big data trends so quickly, realtime stream data processing has become very important. Stream data is a type of big, fast, and unreliable dataset that cannot be handled well by traditional algorithms....
详细信息
ISBN:
(纸本)9781665487689
With the rise of big data trends so quickly, realtime stream data processing has become very important. Stream data is a type of big, fast, and unreliable dataset that cannot be handled well by traditional algorithms. Designing the algorithm that can efficiently process streaming data is a challenging task. This paper shows how important is to make a real-time clustering algorithm for data streams with high concept drift and an algorithm that can adapt to different dimensions. We propose Scalable Random Sampling Online Optimization Weighted Fuzzy c-Means (SRSOO-WFCM) algorithms for handling Big Data in a High-Performance Computing (HPC) environment using an Apache Spark cluster. To compare SRSOO-WFCM with the traditional Online Fuzzy c-Means (OFCM) algorithm, we made a scalable version of OFCM named SOFCM. The proposed SRSOO-WFCM and SOFCM are incrementalalgorithms that involve making one sequential run through the data subsets. We employ both loadable and very large datasets to perform extensive experiments that facilitate comparing the proposed SRSOO-WFCM and SOFCM algorithms. The proposed SRSOOWFCM algorithm performs better than the SOFCM in terms of Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and F-score, respectively.
暂无评论