检索结果-内蒙古大学图书馆

distributed and scalable Sybil identification based on nearest neighbour approximation using big data analysis techniques

引用

CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2019年第6-Sup期22卷 14461-14476页

作者： Valliyammai, Chinnaiah Devakunchari, Ramalingam Anna Univ Fac Informat & Commun Dept Comp Technol MIT Campus Chennai Tamil Nadu India

The problem of Sybil detection has been examined in multiple social media sources like Twitter, LinkedIn and Facebook. The detection of Sybils (fake accounts or social bots) across online social networks emerged as a major challenge due to the current improvement of different social networks, which are promptly generating a very huge data sets termed as big data. The open-source framework, spark-based distributed, fast and scalable nearest neighbor search (S-DFS-NNS) is proposed for profile-based fake account detection across large-scale online social networks. The proposed work performs an efficient parallel processing of the NN search problem. The performance of the k-nearest neighbor (k-NN) search significantly degrades for huge data sets, because the job is computationally hard. The framework is fast and adaptable to expansive, large-scale situations. By using in-memory computation, the suspected users are identified based on the novel private feature. The Spark-DFS-NN search technique provides a substantial performance development over the nearest neighbor computation in large-scale networks. The proposed framework is evaluated using detection accuracy which is able to expose and block a large fraction of suspicious accounts during account creation. The proposed S-DFS-NN framework maintains an approximately consistent and similar performance of 89-95% on the increase of attacks with a latency of 58 ms.

关键词： Sybil attack Online social networks Big data In-memory resilient distributed dataset

来源：评论

学校读者我要写书评

暂无评论

distributed graph cube generation using Spark framework

引用

JOURNAL OF SUPERCOMPUTING 2020年第10期76卷 8118-8139页

作者： Kang, Seok Lee, Suan Kim, Jinho Kangwon Natl Univ Dept Comp Sci Chunchon Kangwon South Korea

Graph OLAP is a technology that generates aggregates or summaries of a large-scale graph based on the properties (or dimensions) associated with its nodes and edges, and in turn enables interactive analyses of the statistical information contained in the graph. To efficiently support these OLAP functions, a graph cube is widely used, which maintains aggregate graphs for all dimensions of the source graph. However, computing the graph cube for a large graph requires an enormous amount of time. While previous approaches have used the MapReduce framework to cut down on this computation time, the recently developed Spark environment offers superior computational performance. To leverage the advantages of Spark, we propose the GraphNaive and GraphTDC algorithms. GraphNaive sequentially computes graph cuboids for all dimensions in a graph, while GraphTDC computes them after first creating an execution plan. We also propose the Generate Multi-Dimension Table method to efficiently create a multidimensional graph table to express the graph. Evaluation experiments demonstrated that the GraphTDC algorithm significantly outperformed Spark SQL's built-in library DataFrame, as the size of graphs increased.

关键词： distributed parallel processing Spark framework resilient distributed dataset Graph cube Data cube Online analytical processing

来源：评论

学校读者我要写书评

暂无评论

Scalable distributed data cube computation for large-scale multidimensional data analysis on a Spark cluster

引用

CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS 2019年第1-Sup期22卷 2063-2087页

作者： Lee, Suan Kang, Seok Kim, Jinho Yu, Eun Jung Kangwon Natl Univ Chuncheon Si Gangwon Do South Korea Kangwon Natl Univ Dept Comp Sci Coll Informat Technol Chuncheon Si Gangwon Do South Korea Yonsei Univ Seoul South Korea

A data cube is a powerful analytical tool that stores all aggregate values over a set of dimensions. It provides users with a simple and efficient means of performing complex data analysis while assisting in decision making. Since the computation time for building a data cube is very large, however, efficient methods for reducing the data cube computation time are needed. Previous works have developed various algorithms for efficiently generating data cubes using MapReduce, which is a large-scale distributed parallel processing framework. However, MapReduce incurs the overhead of disk I/Os and network traffic. To overcome these MapReduce limitations, Spark was recently proposed as a memory-based parallel/distributed processing framework. It has attracted considerable research attention owing to its high performance. In this paper, we propose two algorithms for efficiently building data cubes. The algorithms fully leverage Spark's mechanisms and properties: resilient distributed Top-Down Computation (RDTDC) and resilient distributed Bottom-Up Computation (RDBUC). The former is an algorithm for computing the components (i.e., cuboids) of a data cube in a top-down approach;the latter is a bottom-up approach. The RDTDC algorithm has three key functions. (1) It approximates the size of the cuboid using the cardinality without additional Spark action computation to determine the size of each cuboid during top-down computation. Thus, one cuboid can be computed from the upper cuboid of a smaller size. (2) It creates an execution plan that is optimized to input the smaller sized cuboid. (3) Lastly, it uses a method of reusing the result of the already computed cuboid by top-down computation and simultaneously computes the cuboid of several dimensions. In addition, we propose the RDBUC bottom-up algorithm in Spark, which is widely used in computing Iceberg cubes to maintain only cells satisfying a certain condition of minimum support. This algorithm incorporates two primary s

关键词： distributed processing Spark framework resilient distributed dataset Data warehousing On-line analytical processing Multidimensional data cube Iceberg cube

来源：评论

学校读者我要写书评

暂无评论

Apache Spark based distributed Self-Organizing Map Algorithm for Sensor Data Analysis 43

Apache Spark based Distributed Self-Organizing Map Algorithm...

引用

43rd Annual Conference of the IEEE-Industrial-Electronics-Society (IECON)

作者： Jayaratne, Madhura Alahakoon, Damminda De Silva, Daswin Yu, Xinghuo La Trobe Univ Res Ctr Data Analyt & Cognit Bundoora Vic Australia RMIT Univ Sch Engn Melbourne Vic Australia

ISBN: (纸本)9781538611272

The proliferation of Internets of Things (IoT) technologies in both industrial and non-industrial settings has led to the accumulation of Big Data sets. Analysis of these high volume, high-velocity datasets require advanced processing techniques that incorporate parallel and distributed computations. In this paper, we present a novel distributed self adaptive neural-network algorithm, the distributed Growing Self-Organizing Map (DGSOM) algorithm to address the growing need for unsupervised machine learning of Big Data sets on distributed computing environments. The algorithm was tested on a Big Data set of sensor recordings of human activity collected from wearable devices, 2.8 million records. Results indicate that the distributed algorithm significantly reduces execution time compared to its serial counterpart. Moreover, the self-adaptive nature and controlled growth of the algorithm demonstrates data-driven structure adaptation and multi granular pattern analysis. Overall, the proposed algorithm addresses the need for pattern discovery and visualization from Big Data sets generated by IoT devices which are increasingly commonplace in industrial scenarios.

关键词： Growing Self-Organizing Maps resilient distributed dataset Unsupervised Machine Learning Internet of Things Industrial IoT Big Data analysis

来源：评论

学校读者我要写书评

暂无评论

A Unified Cloud Platform for Autonomous Driving

引用

COMPUTER 2017年第12期50卷 42-49页

作者： Liu, Shaoshan Tang, Jie Wang, Chao Wang, Quan Gaudiot, Jean-Luc PerceptIn Sunnyvale CA 94085 USA South China Univ Technol Sch Comp Sci & Engn Guangzhou Guangdong Peoples R China Baidu Autonomous Driving Unit Beijing Peoples R China Univ Calif Irvine Dept Elect Engn & Comp Sci Irvine CA USA

Tailoring cloud support for each autonomous-driving application would require maintaining multiple infrastructures, potentially resulting in low resource utilization, low performance, and high management overhead. To address this problem, the authors present a unified cloud infrastructure with Spark for distributed computing, Alluxio for distributed storage, and OpenCL to exploit heterogeneous computing resources for enhanced performance and energy efficiency.

关键词： Automobiles Cloud Computing Control Engineering Computing Energy Efficiency Heterogeneous Computing Resources Open CL distributed Storage Alluxio distributed Computing Spark Unified Cloud Infrastructure Autonomous Driving Unified Cloud Platform Autonomous Automobiles Field Programmable Gate Arrays distributed Computing Cloud Computing Computational Modeling Autonomous Vehicles Self Driving Cars distributed Computing distributed Storage Heterogeneous Computing Simulation Testing Model Training Map Generation Hadoop Map Reduce Spark Alluxio Open CL YARN Linux Containers LXC Hadoop distributed File System HDFS Robot Operating System ROS resilient distributed dataset RDD Bin Pipe RDD Parallel distributed Deep Learning Paddle Baidu Convolution Neural Network CNN Li DAR

来源：评论

学校读者我要写书评

暂无评论

Probabilistic modeling of renewable energy source based on Spark platform with large-scale sample data

引用

INTERNATIONAL TRANSACTIONS ON ELECTRICAL ENERGY SYSTEMS 2019年第3期29卷

作者： Yang, Yan Yu, Juan Yang, Mengfan Ren, Pengling Yang, Zhifang Wang, Guisheng Chongqing Univ Key Lab Power Transmiss Equipment & Syst Secur & Chongqing 400030 Peoples R China Johnson Controls Inc Core Platforms Milwaukee WI 53201 USA Chongqing ZENO Big Data Anal Co Ltd Chongqing Peoples R China

In this paper, the probabilistic modeling based on resilient distribution dataset (RDD) of Spark platform is proposed to efficiently process the large-scale sample data of renewable energy source (RES). Based on Spark and Hadoop distributed file system, a parallel and distributed framework compatible with on-hand RES data storage systems is firstly designed for the fast probabilistic modeling of RES. On the basis of the designed framework, a novel parallel estimation algorithm of Wakeby distribution as well as kernel density estimation is developed based on RDD. With the in-memory parallel computing and fault-tolerant characteristics of RDD, the proposed algorithms significantly enhance the parallel execution performance of probabilistic. Besides, the approximate analytical relationship among time consumptions of the proposed algorithms, two important adjustable parameters (degree of parallelism and the number of partitions) of Spark platform, and large sample size of RES is derived, which is helpful for prediction of computational time, hardware configuration setting, and program tuning in the Spark platform. Simulation results with sample size ranging from 7.3 x 10(6) to 3.6 x 10(9) demonstrate the correctness and effectiveness of the proposed techniques.

关键词： renewable energy source large-scale data probabilistic modeling Spark platform resilient distributed dataset

来源：评论

学校读者我要写书评

暂无评论

Scalable formal concept analysis algorithms for large datasets using Spark

引用

JOURNAL OF AMBIENT INTELLIGENCE AND HUMANIZED COMPUTING 2019年第11期10卷 4283-4303页

作者： Chunduri, Raghavendra K. Cherukuri, Aswani Kumar VIT Univ Sch Informat Technol Vellore Tamil Nadu India

In the process of knowledge discovery and representation in large datasets using formal concept analysis, complexity plays a major role in identifying the formal concepts and constructing the concept lattice (digraph of the concepts). For identifying the formal concepts and constructing the digraph from the identified concepts in large datasets, various distributed algorithms are available. However, the existing distributed algorithms are not well suited for concept generation, because the generation of concepts is an iterative process. Existing algorithms are implemented using distributed frameworks like MapReduce and Open MP. These frameworks are not appropriate for iterative applications. Hence, there is a need for efficient distributed algorithms for both formal concept generation and concept lattice digraph construction in large formal contexts. In this paper, we present efficient algorithms using Apache Spark. The various performance metrics used in evaluation prove that the proposed algorithms are more efficient for concept generation and lattice graph construction than existing algorithms.

关键词： Apache Spark Concept lattice Formal concept analysis Hadoop MapReduce resilient distributed dataset

来源：评论

学校读者我要写书评

暂无评论

Scalable algorithm for generation of attribute implication base using FP-growth and spark

引用

SOFT COMPUTING 2021年第14期25卷 9219-9240页

作者： Chunduri, Raghavendra Kumar Cherukuri, Aswani Kumar Vellore Inst Technol Sch Informat Technol & Engn Vellore 632014 Tamil Nadu India

Formal concept analysis (FCA) is an unsupervised machine learning technique used for knowledge discovery and representation. A major task in FCA is the enumeration of the implications to construct the implication base. Even though there are many efficient classical and parallel algorithms proposed for constructing the implication base, the existing algorithms are not well suited for large formal contexts because of their architectural complexity. All the existing works use either stem-base or proper-premise approach to find the implication base in exponential time. Hence, we introduce a distributed algorithm to find the implication base quickly in larger datasets in polynomial time. In this paper, we propose a scalable algorithm to find the implication base using machine learning technique FP-growth, big data processing framework Apache Spark and executed on large formal contexts. Extensive experiments on the real-world datasets show that the proposed algorithm has an improved gain in performance metrics such as execution time, CPU and memory usage. The statistical validations on the experimental results prove that the proposed algorithm has the better potential to find the implication base.

关键词： Apache Spark Data frames Formal concept analysis FP-growth Implication base Machine learning resilient distributed dataset

来源：评论

学校读者我要写书评

暂无评论

Efficient Approach of Analyzing and Generating Intrinsic Information from Weblog

引用

NATIONAL ACADEMY SCIENCE LETTERS-INDIA 2021年第6期44卷 525-527页

作者： Bakariya, Brijesh IK Gujral Punjab Tech Univ Campus Dept Comp Sci & Engn Hoshiarpur Punjab India

The weblog data have unstructured data, and due to this extraction, the desired information from it is a very challenging task. This kind of data is rapidly growing with huge volume. In this paper, the weblog analysis through Pyspark (WAP) algorithm is proposed to analyze the complete weblog dataset and extract useful information from it. In the proposed approach is used resilient distributed dataset (RDD) for running and operating on multiple nodes to do parallel processing on a cluster;for achieving these concepts is used apache pyspark and implemented on jupyter notebook. WAP extracts different results such as host count, path count, status count, etc. This approach also generates patterns and track user behavior. The recommendation is also possible by the proposed algorithm based on user behaviors.

关键词： Weblog Pyspark resilient distributed dataset Jupyter notebook Parallel processing

来源：评论

学校读者我要写书评

暂无评论

Product Pre-Launch Prediction From resilient distributed e-WOM Data

引用

IEEE ACCESS 2020年 8卷 167887-167899页

作者： Narayanan, Sandhya Samuel, Philip Chacko, Mariamma Cochin Univ Sci & Technol Div Informat Technol Kochi 682022 Kerala India Cochin Univ Sci & Technol Dept Comp Sci Kochi 682022 Kerala India Cochin Univ Sci & Technol Dept Ship Technol Kochi 682022 Kerala India

Pre-launch success prediction of a product is a challenge in today's electronic world. Based on this prediction, industries can avoid huge losses by deciding on whether to launch or not to launch a product into the market. We have implemented a Multithreaded Hash join resilient distributed dataset (MHRDD) with a prediction classifier for pre-launch prediction. MHRDD helps to remove the redundancy in the input dataset and improves the performance of the prediction model. Large volume of e-Word of Mouth (e-WOM) data like product reviews, comments and ratings available on internet about products can be used for pre-launch product prediction. In MHRDD, to identify features a distance similarity score is used. In order to remove duplicates, a hash key and join operations are used to create a hash table of significant features. With in-memory computations and hashing on the join operations, this model reduces redundancy of data. This model is scalable and can handle large datasets with good prediction accuracy. This paper presents a novel big data processing method that predicts product success before its launch in the market. Proposed method helps to identify features that are significant for the product to be successful. Based on the pre-launch prediction, companies can reduce cost, effort and time with improved product success.

关键词： Feature extraction Predictive models Computational modeling Redundancy Data models Microsoft Windows Mouth Big data analytics product pre-launch prediction resilient distributed dataset redundancy elimination

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：