检索结果-内蒙古大学图书馆

Mining moving object gathering pattern based on resilient distributed datasets and R-tree index

NEUROCOMPUTING 2020年 393卷 194-202页

作者： He, Qian Chen, Yiting Dong, Qinghe Wang, Yong Guilin Univ Elect Technol Key Lab Cognit Radio & Informat Proc Minist Educ Guilin 541004 Peoples R China Guilin Univ Elect Technol Guangxi Univ Key Lab Cloud Comp & Complex Syst Guilin 541004 Peoples R China

It is important to mine moving object gathering pattern, but the traditional algorithms cannot effectively analyze the trajectory data with mass scale. A parallel mining algorithm based on resilient distributed datasets (RDD-Gathering) and R-tree index is proposed to discover the gathering pattern in the massive trajectory data, and the mining functions are provided as cloud services on Spark. MongoDB cluster is constructed to store massive trajectory data, R-tree index structure is used to store the trajectory data, and the RDD-Gathering algorithm interface is encapsulated with RESTful cloud service. The RDD-Gathering algorithm is implemented based on Apache Spark and performances are better than normal methods in real trajectory dataset testing. (C) 2019 Elsevier B.V. All rights reserved.

关键词： Gathering pattern resilient distributed datasets MongoDB cluster Cloud service

来源：评论

学校读者我要写书评

暂无评论

DBSCAN on resilient distributed datasets 13

DBSCAN on Resilient Distributed Datasets

引用

International Conference on High Performance Computing and Simulation (HPCS)

作者： Cordova, Irving Moh, Teng-Sheng San Jose State Univ Dept Comp Sci San Jose CA 95192 USA

ISBN: (纸本)9781467378130

DBSCAN is a well-known density-based data clustering algorithm that is widely used due to its ability to find arbitrarily shaped clusters in noisy data. However, DBSCAN is hard to scale which limits its utility when working with large data sets. resilient distributed datasets (RDDs), on the other hand, are a fast data-processing abstraction created explicitly for in-memory computation of large data sets. This paper presents a new algorithm based on DBSCAN using the resilient distributed datasets approach: RDD-DBSCAN. RDD-DBSCAN overcomes the scalability limitations of the traditional DBSCAN algorithm by operating in a fully distributed fashion. The paper also evaluates an implementation of RDD-DBSCAN using Apache Spark, the official RDD implementation.

关键词： DBSCAN Apache Spark data clustering parallel systems data partition resilient distributed datasets MapReduce

来源：评论

学校读者我要写书评

暂无评论

A Parallel Version of Differential Evolution Based on resilient distributed datasets Model 1

引用

10th International Conference on Bio-Inspired Computing - Theories and Applications (BIC-TA)

作者： Deng, Changshou Tan, Xujie Dong, Xiaogang Tan, Yucheng JiuJiang Univ Sch Informat Sci & Technol Jiujiang 332005 Jiangxi Peoples R China

ISBN: (数字)9783662490143

ISBN: (纸本)9783662490143;9783662490136

MapReduce is a popular cloud computing platform which has been widely applied in large-scale data-intensive fields. However, when dealing with computation extensive tasks, particularly, iterative computation, frequent loading Map and Reduce processes will lead to overhead. resilient distributed datasets model which has been implemented in Spark, is an in-memory clustering computing which can overcome this shortcoming efficiently. In this paper, we attempt to use resilient distributed datasets model to parallelize Differential Evolution algorithm. A wide range of benchmark problems have been adopted to conduct numerical experiment, and the speedup of PDE due to use of resilient distributed datasets model is demonstrated. The results show us that resilient distributed datasets model is a potential way to parallelize evolutionary algorithm.

关键词： Parallel differential evolution Spark resilient distributed datasets Transformation operation Action operation

来源：评论

学校读者我要写书评

暂无评论

distributed cache strategy based on LT codes under spark platform

引用

JOURNAL OF SUPERCOMPUTING 2024年第11期80卷 16519-16545页

作者： Shang, Jing Zhang, Yifei Wang, Jibin Wu, Zhihui Xiao, Zhiwen China Mobile Informat Technol Ctr 2 Guangningbo St Beijing 100032 Peoples R China

During the execution of tasks in a distributed computing system, the master needs to assign new workers to replace lagging workers that usually occur due to various issues such as network problems, hardware failures, software bugs, or high cluster load. If the completed part of tasks by lagging workers is not cached timely, new workers would need to recalculate it, leading to further delays. In the Spark platform, checkpointing allows for the preservation of intermediate results computed by workers in resilient distributed datasets (RDDs) by storing them in Hadoop distributed file system (HDFS). However, storing and retrieving data from HDFS still remains time-consuming. Traditional distributed cache strategies neglect to utilize both coding techniques and different lag probabilities of workers to reduce overhead of caching data. This paper introduces a distributed cache strategy based on Luby transform (LT) codes to enhance system's effectiveness and robustness. An optimization method is designed within prescribed cache limits to guide the selection of RDDs for caching. Additionally, a novel algorithm is proposed for determining the participation of specific partitions in encoding caching under consideration of worker failure probabilities and overhead of encoding and decoding. In the event of worker failure, intermediate data can be swiftly recovered through decoding, and then the standbys can rapidly take over stragglers' tasks, hence markedly mitigating overall task latency. Theoretical analysis and experimental results demonstrate that the LT code-based caching strategy exhibits more efficiency and lower latency compared to recomputation, HDFS-based and weight-based caching methods, and it saves much more memory overhead compared to Redis.

关键词： Spark platform resilient distributed datasets Coded distributed caching Luby transform codes Degree distribution

来源：评论

学校读者我要写书评

暂无评论

Online Payment Fraud Detection for Big Data 20th

Online Payment Fraud Detection for Big Data

引用

20th International Conference on distributed Computing and Intelligent Technology (ICDCIT)

作者： Tawde, Samiksha Dattaprasad Arora, Sandhya Thakur, Yashasvee Shitalkumar MKSSSs Cummins Coll Engn Women Dept Comp Engn Pune Maharashtra India

ISBN: (纸本)9783031505829;9783031505836

Modern economic life is now greatly facilitated by online payment systems, which allow for seamless financial transactions. However, the risk of online payment fraud has greatly increased along with the growth of digital transactions. This calls for the creation of sophisticated fraud detection systems that can instantly evaluate huge amounts of transaction data. This study suggests a novel method for identifying online payment fraud by utilizing big data management techniques, more specifically PySpark's capabilities. PySpark uses resilient distributed datasets (RDD), data structure and stores data in RAM instead of writing it to disk after each operation. RDD operations are lazy i.e., they will not execute unless an action operation is called on them. After preprocessing the data Machine Learning algorithms from Spark ML package are applied, the ML library of PySpark provides optimized Machine Learning capabilities for Classification problems that require distributed computing. Further, models of classification algorithms that qualify with the best metrics are developed on our dataset and used for making accurate detections. Our Fraud detection system aims to assist Large organizations in assessing their enormous amount of transaction data to detect possible anomalies or fraudulent activities.

关键词： PySpark Fraud Detection Big Data resilient distributed datasets Decision Tree Random Forest Logistic Regression

来源：评论

学校读者我要写书评

暂无评论

Big data and Spark: Comparison with Hadoop

Big data and Spark: Comparison with Hadoop

引用

4th World Conference on Smart Trends in Systems, Security and Sustainability (WorldS4)

作者： Benlachmi, Yassine Hasnaoui, Moulay Lahcen ENSAM Moulay Ismail Univ LMMI LAb Meknes Morocco

ISBN: (纸本)9781728168234

Now a days, data is generating by different industries on a daily basis which amount growing from terabyte to petabyte. To handle this large amount of data, big data technology. is great revolution and has impacted the trends of the applied science. For the analysis of the large amount of data, Hadoop MapReduce technology is used in parallel multiple nodes. The numerous amount of data and information is stored through HDFS, Map and Reduce are two main functions of the MapReduce. There are many loop holes in MapReduce so for fast queries and real time data stream, Spark is introduced. The base of the Spark is DAG and RDD approaches. The main purpose of this research is to make comparison between the basic features of Hadoop and Spark on which basis their performance will be evaluate.

关键词： Big Data resilient distributed datasets MapReduce Spark DAG HDFS Hadoop

来源：评论

学校读者我要写书评

暂无评论

A Hybrid Mechanism of Particle Swarm Optimization and Differential Evolution Algorithms based on Spark

引用

KSII TRANSACTIONS ON INTERNET AND INFORMATION SYSTEMS 2019年第12期13卷 5972-5989页

作者： Fan, Debin Lee, Jaewan Jiujiang Univ Sch Informat Sci & Technol Jiujiang Peoples R China Kunsan Natl Univ Dept Informat & Commun Engn Kunsan South Korea

With the onset of the big data age, data is growing exponentially, and the issue of how to optimize large-scale data processing is especially significant. Large-scale global optimization (LSGO) is a research topic with great interest in academia and industry. Spark is a popular cloud computing framework that can cluster large-scale data, and it can effectively support the functions of iterative calculation through resilient distributed datasets (RDD). In this paper, we propose a hybrid mechanism of particle swarm optimization (PSO) and differential evolution (DE) algorithms based on Spark (SparkPSODE). The SparkPSODE algorithm is a parallel algorithm, in which the RDD and island models are employed. The island model is used to divide the global population into several subpopulations, which are applied to reduce the computational time by corresponding to RDD's partitions. To preserve population diversity and avoid premature convergence, the evolutionary strategy of DE is integrated into SparkPSODE. Finally, SparkPSODE is conducted on a set of benchmark problems on LSGO and show that, in comparison with several algorithms, the proposed SparkPSODE algorithm obtains better optimization performance through experimental results.

关键词： Particle Swarm Optimization Differential Evolution Large-scale optimization Cloud Computing resilient distributed datasets

来源：评论

学校读者我要写书评

暂无评论

Apache Spark: A Big Data Processing Engine 2

Apache Spark: A Big Data Processing Engine

引用

2nd IEEE Middle East and North Africa COMMunications Conference (IEEE MENACOMM)

作者： Shaikh, Eman Mohiuddin, Iman Alufaisan, Yasmeen Nahvi, Irum Prince Mohammad Bin Fahd Univ Coll Comp Engn & Sci Khobar Saudi Arabia

ISBN: (纸本)9781728136875

Big data analysis has influenced the industry market. It has a significant impact on large and varied datasets to exhibit the hidden patterns and other revelations. Apache Hadoop, Apache Flink and Apache Storm are some commonly used frameworks for big data analysis. Apache Spark is a consolidated big data analytics engine and provides absolute data parallelism. This paper scrutinizes a technical review on big data analytics using Apache Spark and how it uses in-memory computation that makes it remarkably faster as compared to other corresponding frameworks. Moreover, Spark also provides exceptional batch processing and stream processing capabilities. Furthermore, it also discuses over the multithreading and con-currency capabilities of Apache Spark. The point of convergence is architecture, hardware requirements, ecosystem, use cases, features of Apache Spark and the use of Spark in emerging technologies.

关键词： Big Data Analytics Machine Learning Stream Processing resilient distributed datasets

来源：评论

学校读者我要写书评

暂无评论

A distributed Approach for Constructing Generalized Suffix Tree on Spark by Using Optimized Elastic Range Algorithm 5

A Distributed Approach for Constructing Generalized Suffix T...

引用

5th International Conference on Advanced Cloud and Big Data (CBD)

作者： Li, Yunhao Jin, Jiahui Xiong, Runqun Luo, Junzhou Southeast Univ Sch Comp Sci & Engn Nanjing Jiangsu Peoples R China

ISBN: (纸本)9781538610725

The generalized suffix tree (GST) is a tree structure that is widely used by string-based applications such as DNA sequence pattern search, data compression and time series analysis. It can efficiently accelerate the string operations like matching approximate strings and finding the longest common substring. In the big data era, the applications processing large-scale strings (e.g., genomic sequences) are common, so it is important to design scalable approaches for constructing massive GSTs. In this paper, we introduce a distributed approach for constructing GSTs on top of Apache Spark, a general-purpose big data processing system. The framework of our approach is based on the Elastic Range Algorithm (ERA), a state-of-the-art GST construction algorithm. Different from the original ERA, our approach optimizes the structure of GST and the subtree construction, which greatly reduces the memory requirement of GST construction and storage. In addition, we propose serval optimization techniques to speed up our approach. Our experimental results show that our approach can index billion-symbol strings within 5 minutes on a 8-worker Spark cluster. Moreover, the optimization techniques get about 5x speedup on the overall indexing time.

关键词： Generalized Suffix Tree Spark resilient distributed datasets String Matching

来源：评论

学校读者我要写书评

暂无评论

Streamlining the genomics processing pipeline via named pipes and persistent Spark datasets 17

Streamlining the genomics processing pipeline via named pipe...

引用

17th IEEE International Conference on Bioinformatics and Bioengineering (BIBE)

作者： Blair, Walter Joao, Leonardo de Melo Davis, Larry Anderson, Paul Coll Charleston Dept Comp Sci Charleston SC 29401 USA Pontificia Univ Catolica Minas Gerais Dept Comp Sci Campus Pocos De Caldas Brazil Western Washington Univ Dept Comp Sci Bellingham WA 98225 USA

ISBN: (纸本)9781538613245

In this paper we investigate the use of Unix named pipes and an in-memory datagrid to reduce the I/O requirements of conventional and exploratory genomics processing pipelines. Apache Spark provides an in-memory framework for distributed computational genomics that has realized significant improvements over conventional pipelines in speed and flexibility. Even in the Spark framework, however, pipeline components create I/O bottlenecks by reading and writing intermediate files that are later discarded. Apache Ignite provides a framework for persisting a Spark dataset in memory between modular pipeline applications, and Unix named pipes have long provided a mechanism by which data can be transferred in-memory. We compared the runtime performance of a standard genomics pipeline that transmits Spark data using named pipes and/or Ignite's in-memory datagrid. Our results demonstrate that Ignite can improve the runtime performance of in-memory RDD actions and that keeping pipeline components in memory with Ignite and named pipes eliminates a major I/O bottleneck.

关键词： genomics apache spark apache ignite pipelines resilient distributed datasets

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：