检索结果-内蒙古大学图书馆

The Parallel Fuzzy C-Median Clustering Algorithm Using Spark for the big data

IEEE ACCESS 2024年 12卷 151785-151804页

作者： Alam Mallik, Moksud Fariza Zulkurnain, Nurul Siddiqui, Sumrana Sarkar, Rashel Int Islamic Univ Malaysia Dept Elect & Comp Engn Kuala Lumpur 53100 Malaysia Lords Inst Engn & Technol Dept Comp Sci & Engn Hyderabad 500091 India Deccan Coll Engn & Technol Dept Comp Sci & Engn Hyderabad 500001 India Assam Royal Global Univ Dept Comp Sci & Engn Gauhati 781035 Assam India

big data for sustainable development is a global issue due to the explosive growth of data and according to the forecasting of International data Corporation(IDC), the amount of data in the world will double every 18 months, and the Global data-sphere is expected to more than double in size from 2022 to 2026. The analysis, processing, and storing of big data is a challenging research concern due to data imperfection, massive data size, computational difficulty, and lengthy evaluation time. Clustering is a fundamental technique in data analysis and data mining, and it becomes particularly challenging when dealing with big data due to the sheer volume, velocity, and variety of the data. big data frameworks like Hadoop MapReduce and Spark are potent tools that provide an effective way to analyze huge datasets that are being processed by the Hadoop cluster. Apache Spark is one of the most widely used large-scale data processing engines due to its speed, low latency in-memory computing, and powerful analytics. Therefore, we develop a Parallel Fuzzy C-Median Clustering Algorithm Using Spark for big data that can handle large datasets while maintaining high accuracy and scalability. The algorithm employs a distance-based clustering approach to determine the similarity between data points and group them in combination with sampling and partitioning techniques. In the sampling phase, a representative subset of the dataset is selected. In the partitioning phase, the data is partitioned into smaller subsets that can be clustered in parallel across multiple nodes. The suggested method, implemented in the databricks cloud platform provides high clustering accuracy, as measured by clustering evaluation metrics such as the silhouette coefficient, cost function, partition index, clustering entropy. The experimental results show that c=5, which is consistent for cost function with the ideal silhouette coefficient of 1, is the optimal number of clusters for this dataset. A comparativ

关键词： Clustering algorithms big data Partitioning algorithms Sparks Measurement Cluster computing Social networking (online) Fuzzy systems data clustering big data framework fuzzy C means fuzzy C median spark

来源：评论

学校读者我要写书评

暂无评论

DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning Approach for big data frameworks 22

DeepCAT: A Cost-Efficient Online Configuration Auto-Tuning A...

引用

Proceedings of the 51st International Conference on Parallel Processing

作者： Hui Dou Yilun Wang Yiwen Zhang Pengfei Chen School of Computer Science and Technology Anhui University China School of Computer Science and Engineering Sun Yat-sen University China

ISBN: (纸本)9781450397339

To support different application scenarios, big data frameworks usually provide a large number of performance-related configuration parameters. Online auto-tuning these parameters based on deep reinforcement learning to achieve a better performance has shown their advantages over search-based and machine learning-based approaches. Unfortunately, the time consumption during the online tuning phase of conventional DRL-based methods is still heavy, especially for big data applications. Therefore, in this paper, we propose DeepCAT, a cost-efficient deep reinforcement learning-based approach to achieve online configuration auto-tuning for big data frameworks. To reduce the total online tuning cost: 1) DeepCAT utilizes the TD3 algorithm instead of DDPG to alleviate value overestimation; 2) DeepCAT modifies the conventional experience replay to fully utilize the rare but valuable transitions via a novel reward-driven prioritized experience replay mechanism; 3) DeepCAT designs a Twin-Q Optimizer to estimate the execution time of each action without the costly configuration evaluation and optimize the sub-optimal ones to achieve a low-cost exploration-exploitation trade off. Experimental results based on a local 3-node Spark cluster and HiBench benchmark applications show that DeepCAT is able to speed up the best execution time by a factor of 1.45 × and 1.65 × on average respectively over CDBTune and OtterTune, while consuming up to 50.08% and 53.39% less total tuning time.

关键词： Deep Reinforcement Learning big data framework Performance Optimization Online Configuration Tuning

来源：评论

学校读者我要写书评

暂无评论

Top-k dominating queries on incomplete large dataset

引用

JOURNAL OF SUPERCOMPUTING 2022年第3期78卷 3976-3997页

作者： Wu, Jimmy Ming-Tai Wei, Min Wu, Mu-En Tayeb, Shahab Shandong Univ Sci & Technol Coll Comp Sci & Engn Qingdao Peoples R China Natl Taipei Univ Technol Dept Informat & Finance Management Taipei Taiwan Calif State Univ Fresno Dept Elect & Comp Engn Fresno CA 93740 USA

Top-k dominating (TKD) query is one of the methods to find the interesting objects by returning the k objects that dominate other objects in a given dataset. Incomplete datasets have missing values in uncertain dimensions, so it is difficult to obtain useful information with traditional data mining methods on complete data. BitMap Index Guided Algorithm (big) is a good choice for solving this problem. However, it is even harder to find top-k dominance objects on incomplete big data. When the dataset is too large, the requirements for the feasibility and performance of the algorithm will become very high. In this paper, we proposed an algorithm to apply MapReduce on the whole process with a pruning strategy, called Efficient Hadoop BitMap Index Guided Algorithm (EHbig). This algorithm can realize TKD query on incomplete datasets through BitMap Index and use MapReduce architecture to make TKD query possible on large datasets. By using the pruning strategy, the runtime and memory usage are greatly reduced. What's more, we also proposed an improved version of EHbig (denoted as IEHbig) which optimizes the whole algorithm flow. Our in-depth work in this article culminates with some experimental results that clearly show that our proposed algorithm can perform well on TKD query in an incomplete large dataset and shows great performance in a Hadoop computing cluster.

关键词： Top-textitk dominating query Incomplete data Dominance relationship big data framework MapReduce

来源：评论

学校读者我要写书评

暂无评论

A Distributed System for Optimal Scale Feature Extraction and Semantic Classification of Large-Scale Airborne LiDAR Point Clouds 17th

A Distributed System for Optimal Scale Feature Extraction an...

引用

17th International Conference on Distributed Computing and Intelligent Technology

作者： Singh, Satendra Sreevalsan-Nair, Jaya Int Inst Informat Technol Graph Visualizat Comp Lab 26-C Elect City Bangalore 560100 Karnataka India

ISBN: (纸本)9783030656201;9783030656218

Airborne LiDAR (Light Detection and Ranging) or aerial laser scanning (ALS) technology can capture large-scale point cloud data, which represents the topography of large regions. The raw point clouds need to be managed and processed at scale for exploration and contextual understanding of the topographical data. One of the key processing steps is feature extraction from pointwise local geometric descriptors for object-based classification. The state of the art involves finding an optimal scale for computing the descriptors, determined using descriptors across multiple scales, which becomes computationally intensive in the case of big data. Hence, we propose the use of a widely used big data analytics framework integration of Apache Spark and Cassandra, for extracting features at optimal scale, semantic classification using a random forest classifier, and interactive visualization. The visualization involves real-time updates to the selection of regions of interest, and display of feature vectors upon a change in the computation of descriptors. We show the efficacy of our proposed application through our results in the DALES aerial LiDAR point cloud.

关键词： big data framework Apache Spark Cassandra LiDAR point cloud Multiscale feature extraction Semantic classification

来源：评论

学校读者我要写书评

暂无评论

Towards Developing a Robust Intrusion Detection Model Using Hadoop-Spark and data Augmentation for IoT Networks

引用

SENSORS 2022年第20期22卷 7726页

作者： Sanchez, Ricardo Alejandro Manzano Zaman, Marzia Goel, Nishith Naik, Kshirasagar Joshi, Rohit Cistech Ltd 201-203 Colonnade Rd Nepean ON K2E 7K3 Canada Cistel Technol Inc 30 Concourse Gate Nepean ON K2E 7V7 Canada Univ Waterloo Dept Elect & Comp Engn 200 Univ Ave W Waterloo ON N2L 3G1 Canada

In recent years, anomaly detection and machine learning for intrusion detection systems have been used to detect anomalies on Internet of Things networks. These systems rely on machine and deep learning to improve the detection accuracy. However, the robustness of the model depends on the number of datasamples available, quality of the data, and the distribution of the data classes. In the present paper, we focused specifically on the amount of data and class imbalanced since both parameters are key in IoT due to the fact that network traffic is increasing exponentially. For this reason, we propose a framework that uses a big data methodology with Hadoop-Spark to train and test multi-class and binary classification with one-vs-rest strategy for intrusion detection using the entire BoT IoT dataset. Thus, we evaluate all the algorithms available in Hadoop-Spark in terms of accuracy and processing time. In addition, since the BoT IoT dataset used is highly imbalanced, we also improve the accuracy for detecting minority classes by generating more datasamples using a Conditional Tabular Generative Adversarial Network (CTGAN). In general, our proposed model outperforms other published models including our previous model. Using our proposed methodology, the F1-score of one of the minority class, i.e., Theft attack was improved from 42% to 99%.

关键词： IoT (internet of things) security big data framework imbalaced datasets CTGAN hadoop-spark BoT-IoT

来源：评论

学校读者我要写书评

暂无评论

A DISTRIBUTED SYSTEM FOR MULTISCALE FEATURE EXTRACTION AND SEMANTIC CLASSIFICATION OF LARGE-SCALE LIDAR POINT CLOUDS

A DISTRIBUTED SYSTEM FOR MULTISCALE FEATURE EXTRACTION AND S...

引用

IEEE India Geoscience and Remote Sensing Symposium (InGARSS)

作者： Singh, Satendra Sreevalsan-Nair, Jaya Int Inst Informat Technol Bangalore Graph Visualizat Comp Lab Bangalore 560100 Karnataka India

ISBN: (纸本)9781728131146

Managing and processing large-scale point clouds are much needed for the exploration and contextual understanding of the data. Hence, we explore the use of a widely used big data analytics framework, Apache Spark, in distributed systems for large-scale point cloud processing. To effectively use Spark, we propose to use its integration with Cassandra for persistent storage, and to appropriately partition the point cloud across the nodes in the distributed system. We use this integrated framework for multiscale feature extraction and semantic classification using random forest classifier. We have shown the efficacy of our proposed application through our results in the DALES aerial LiDAR point cloud.

关键词： big data framework Apache Spark Cassandra Aerial LiDAR point cloud multiscale feature extraction semantic classification

来源：评论

学校读者我要写书评

暂无评论

Co-design Implications of Cost-effective On-demand Acceleration for Cloud Healthcare Analytics: The AEGLE approach 22

Co-design Implications of Cost-effective On-demand Accelerat...

引用

22nd Design, Automation and Test in Europe Conference and Exhibition (DATE)

作者： Masouros, Dimosthenis Koliogeorgi, Konstantina Zervakis, Georgios Kosvyra, Alexandra Chytas, Achilleas Xydis, Sotirios Chouvarda, Ioanna Soudris, Dimitrios Natl Tech Univ Athens Microprocessors & Digital Syst Lab ECE Athens Greece Aristotle Univ Thessaloniki Lab Comp Med Informat & Biomed Imaging Technol SM Thessaloniki Greece

ISBN: (纸本)9783981926323

Nowadays, big data and machine learning are transforming the way we realize and manage our data. Even though the healthcare domain has recognized big data analytics as a prominent candidate, it has not yet fully grasped their promising benefits that allow medical information to be converted to useful knowledge. In this paper, we introduce AEGLE's big data infrastructure provided as a Platform as a Service. Utilizing the suite of genomic analytics from the Chronic Lymphocytic Leukaemia (CLL) use case, we show that on-demand acceleration is profitable w.r.t a pure software cloud-based solution. However, we further show that on-demand acceleration is not offered as a "free-lunch" and we provide an in-depth analysis and lessons learnt on the co-design implications to be carefully considered for enabling cost-effective acceleration at the cloud-level.

关键词： Cloud Platform as a Service (PaaS) big data framework big data Analytics Co-design On-demand Acceleration Genomic

来源：评论

学校读者我要写书评

暂无评论

Tweets Sentiment Analysis for Healthcare on big data Processing and IoT Architecture Using Maximum Entropy Classifier 1st

Tweets Sentiment Analysis for Healthcare on Big Data Process...

引用

1st International Conference on big data Analysis and Deep Learning (ICBDL)

作者： Htet, Hein Khaing, Soe Soe Myint, Yi Yi Univ Technol Yatanarpon Cyber City Pyin Oo Lwin Myanmar

ISBN: (纸本)9789811308697;9789811308680

People are too rare to discuss or talk about their health problems with each other and, it is very poor to notice about their realistic health situation. But nowadays, most of the people friendly used social media and people have started expressing their feelings and activities on it. Focus only on Twitter, users' created tweets composed of news, politics, life conversation which can also be applied for doing a variety of analysis purposes. Therefore, healthcare system is developed to mine about the health state of Twitter user and to provide health authorities to easily check about their continental health behavior based on the Twitter data. Maximum Entropy classifier (MaxEnt) is used to perform sentiment analysis on their tweets to suggest their health condition (good, fair, or bad). It is interacting with Twitter data (big data environment) and so, Internet of Things (IoT) based big data processing framework is built to be efficiently handled large amount of Twitter user' data. The aim of this paper is to propose healthcare system using MaxEnt classifier and big data processing using Hadoop framework integrated with Internet of Things architecture.

关键词： Sentiment analysis big data framework IoT

来源：评论

学校读者我要写书评

暂无评论

High-Utility Itemset Mining in big dataset 6

High-Utility Itemset Mining in Big Dataset

引用

IEEE International Conference on Consumer Electronics-Taiwan (IEEE ICCE-TW)

作者： Wu, Jimmy Ming-Tai Lin, Jerry Chun-Wei Chen, Chien-Ming Shandong Univ Sci & Technol Coll Comp Sci & Engn Qingdao Peoples R China Western Norway Univ Appl Sci Dept Comp Math & Phys Bergen Norway

ISBN: (纸本)9781728132792

High-utility mining (HUIM) is an extended concept from frequent itemset mining (FIM). It emphasizes the more important factors, such as profits or the weight of an itemset in commercial applications. In this paper, we assume a dataset is too big to be loaded in the memory, then propose a MapReduce framework to handle this kind of situation, and try to reduce the times of scanning dataset as possible and maximize parallelization of the process.

关键词： HUIM data mining big data framework MapReduce

来源：评论

学校读者我要写书评

暂无评论

Managing food security through food waste and loss: Small data to big data

引用

COMPUTERS & OPERATIONS RESEARCH 2018年 98卷 367-383页

作者： Irani, Zahir Sharif, Amir M. Lee, Habin Aktas, Emel Topaloglu, Zeynep van't Wout, Tamara Huda, Samsul Univ Bradford Bradford W Yorkshire England Brunel Univ Uxbridge Middx England Cranfield Univ Cranfield Beds England Georgetown Univ Ar Rayyan Qatar Western Sydney Univ Penrith NSW Australia

This paper provides a management perspective of organisational factors that contributes to the reduction of food waste through the application of design science principles to explore causal relationships between food distribution (organisational) and consumption (societal) factors. Qualitative data were collected with an organisational perspective from commercial food consumers along with large-scale food importers, distributors, and retailers. Cause-effect models are built and "what-if simulations are conducted through the development and application of a Fuzzy Cognitive Map (FCM) approaches to elucidate dynamic interrelationships. The simulation models developed provide a practical insight into existing and emergent food losses scenarios, suggesting the need for big data sets to allow for generalizable findings to be extrapolated from a more detailed quantitative exercise. This research offers itself as evidence to support policy makers in the development of policies that facilitate interventions to reduce food losses. It also contributes to the literature through sustaining, impacting and potentially improving levels of food security, underpinned by empirically constructed policy models that identify potential behavioural changes. It is the extension of these simulation models set against a backdrop of a proposed big data framework for food security, where this study sets avenues for future research for others to design and construct big data research in food supply chains. This research has therefore sought to provide policymakers with a means to evaluate new and existing policies, whilst also offering a practical basis through which food chains can be made more resilient through the consideration of management practices and policy decisions. (C) 2017 The Author(s). Published by Elsevier Ltd.

关键词： Food security Qatar big data framework Food waste Food loss Fuzzy cognitive map (FCM) Interrelationships Design science

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：