检索结果-内蒙古大学图书馆

Real-time big data processing for anomaly detection: A Survey

INTERNATIONAL JOURNAL OF INFORMATION MANAGEMENT 2019年 45卷 289-307页

作者： Habeeb, Riyaz Ahamed Ariyaluran Nasaruddin, Fariza Gani, Abdullah Hashem, Ibrahim Abaker Targio Ahmed, Ejaz Imran, Muhammad Univ Malaya Fac Comp Sci & Informat Technol Dept Informat Syst Kuala Lumpur 50603 Malaysia Taylors Univ Sch Comp & Informat Technol Subang Jaya 47500 Selangor Malaysia Univ Malaya Ctr Res Mobile Cloud Comp Kuala Lumpur Malaysia King Saud Univ Coll Appl Comp Sci Riyadh Saudi Arabia

The advent of connected devices and omnipresence of Internet have paved way for intruders to attack networks, which leads to cyber-attack, financial loss, information theft in healthcare, and cyber war. Hence, network security analytics has become an important area of concern and has gained intensive attention among researchers, off late, specifically in the domain of anomaly detection in network, which is considered crucial for network security. However, preliminary investigations have revealed that the existing approaches to detect anomalies in network are not effective enough, particularly to detect them in real time. The reason for the inefficacy of current approaches is mainly due the amassment of massive volumes of data though the connected devices. Therefore, it is crucial to propose a framework that effectively handles real time big data processing and detect anomalies in networks. In this regard, this paper attempts to address the issue of detecting anomalies in real time. Respectively, this paper has surveyed the state-of-the-art real-time big data processing technologies related to anomaly detection and the vital characteristics of associated machine learning algorithms. This paper begins with the explanation of essential contexts and taxonomy of real-time big data processing, anomalous detection, and machine learning algorithms, followed by the review of big data processing technologies. Finally, the identified research challenges of real-time big data processing in anomaly detection are discussed.

关键词： Real-time big data processing Anomaly detection and machine learning algorithms

来源：评论

学校读者我要写书评

暂无评论

Low Latency big data processing without Prior Information

引用

IEEE TRANSACTIONS ON CLOUD COMPUTING 2021年第4期9卷 1521-1534页

作者： Hu, Zhiming Li, Baochun Qin, Zheng Goh, Rick Siow Mong Univ Toronto Dept Elect & Comp Engn Toronto ON M5S 2E4 Canada ASTAR Inst High Performance Comp Singapore 138632 Singapore

Job scheduling plays an important role in improving the overall system performance in big data processing frameworks. Simple job scheduling policies, such as Fair and FIFO scheduling, do not consider job sizes and may degrade the performance when jobs of varying sizes arrive. More elaborate job scheduling policies make the convenient assumption that jobs are recurring, and complete information about their sizes is available from their prior runs. In this paper, we design and implement an efficient and practical job scheduler for big data processing systems to achieve better performance even without prior information about job sizes. The superior performance of our job scheduler originates from the design of multiple level priority queues, where jobs are demoted to lower priority queues if the amount of service consumed so far reaches a certain threshold. In this case, jobs in need of a small amount of service can finish in the topmost several levels of queues, while jobs that need a large amount of service to complete are moved to lower priority queues to avoid head-of-line blocking. Our new job scheduler can effectively mimic the shortest job first scheduling policy without knowing the job sizes in advance. To demonstrate its performance, we have implemented our new job scheduler in YARN, a popular resource manager used by Hadoop/Spark, and validated its performance with experiments in both real testbeds including Amazon EC2 and large-scale trace-driven simulations. Our experimental and simulation results have strongly confirmed the effectiveness of our design: our new job scheduler can reduce the average job response time of the Fair scheduler by up to 45 percent and achieve better fairness at the same time.

关键词： Job scheduling big data processing multilevel feedback queue

来源：评论

学校读者我要写书评

暂无评论

A Distributed Multi-Agent System (MAS) Application For continuous and Integrated big data processing 15th

A Distributed Multi-Agent System (MAS) Application For conti...

引用

15th European Conference on Ambient Intelligence (AmI)

作者： Shashaj, Ariona Mastrorilli, Federico Morrelli, Massimiliano Pansini, Giacomo Iannucci, Enrico Polito, Massimiliano Network Contacts Molfetta BA Italy

ISBN: (纸本)9783030342555;9783030342548

With the advent of Ambient Intelligence (AmI) as an interdisciplinary methodology which ranges from Ubiquitous and Pervasive computing to Artificial Intelligence with the final aim to create a sensitive and responsive environment, the focus is moving towards integrated solutions of the encompassed technologies. Multi-Agent System (MAS) approach, characterized by a set of autonomous intelligent agents which can cooperate in order to achieve a common goal, can help the development of AmI integrated solutions. In this paper, we present a distributed MAS environment, Multi-Agent Specialized system (MASs), which supports the development of integrated AmI solutions. An application scenario considering the case of continuous big data processing is shown.

关键词： Multi-Agent system Distributed system big data processing

来源：评论

学校读者我要写书评

暂无评论

Lambda Architecture for Cost-effective Batch and Speed big data processing 3

Lambda Architecture for Cost-effective Batch and Speed Big D...

引用

IEEE International Conference on big data

作者： Kiran, Mariam Murphy, Peter Monga, Inder Dugan, Jon Baveja, Sartaj Singh Univ Bradford Sch Comp Sci Bradford BD7 1DP W Yorkshire England Energy Sci Network ESnet Washington DC USA Netaji Subhas Inst Technol New Delhi India

ISBN: (纸本)9781479999255

Sensor and smart phone technologies present opportunities for data explosion, streaming and collecting from heterogeneous devices every second. Analyzing these large datasets can unlock multiple behaviors previously unknown, and help optimize approaches to city wide applications or societal use cases. However, collecting and handling of these massive datasets presents challenges in how to perform optimized online data analysis 'on-the-fly', as current approaches are often limited by capability, expense and resources. This presents a need for developing new methods for data management particularly using public clouds to minimize cost, network resources and on-demand availability. This paper presents an implementation of the lambda architecture design pattern to construct a data-handling backend on Amazon EC2, providing high throughput, dense and intense data demand delivered as services, minimizing the cost of the network maintenance. This paper combines ideas from database management, cost models, query management and cloud computing to present a general architecture that could be applied in any given scenario where affordable online data processing of big datasets is needed. The results are presented with a case study of processing router sensor data on the current ESnet network data as a working example of the approach. The results showcase a reduction in cost and argue benefits for performing online analysis and anomaly detection for sensor data.

关键词： big data processing lambda architecture Amazon EC2 sensor data analysis

来源：评论

学校读者我要写书评

暂无评论

Analysis of the effect of weather determinants on lodging demands using big data processing 12

Analysis of the effect of weather determinants on lodging de...

引用

12th IEEE/ACS International Conference of Computer Systems and Applications (AICCSA)

作者： Jeon, Seungwoo Hong, Bonghee Im, Hyeongsoon Pusan Natl Univ Dept Elect & Comp Engn Busan South Korea

ISBN: (纸本)9781509004782

Weather conditions determine the variation in the floating population at tourist sites and, consequently, the number of persons who stay overnight;therefore, it is necessary to select those determinants that are most influenced by the weather. Our aim is to construct a multiple linear regression model and system based on big data processing tools to process large volumes of meteorological and population data with the ultimate goal of supporting predicted accommodation congestion with high prediction accuracy. First, we transform the floating population data into accommodation congestion by estimating the resident population during periods of high demand. Second, we conduct a multiple linear regression analysis with variable removal steps and a residual analysis. Moreover, to verify our regression model, we use two prediction accuracy measures: the mean absolute percentage error and the R-squared value. The final part of the paper describes the construction of a big data processing system that was built to compute the series of prediction operations.

关键词： Floating population data Weather determinants Accommodation congestion Multiple linear regression model big data processing

来源：评论

学校读者我要写书评

暂无评论

Composable and Efficient Functional big data processing Framework 3

Composable and Efficient Functional Big Data Processing Fram...

引用

IEEE International Conference on big data

作者： Wu, Dongyao Sakr, Sherif Zhu, Liming Lu, Qinghua NICTA Software Syst Res Grp Sydney NSW Australia Univ New South Wales Sch Comp Sci & Engn Sydney NSW Australia King Saud bin Abdulaziz Univ Hlth Sci Riyadh Saudi Arabia China Univ Petr Coll Comp & Commun Engn Qingdao Peoples R China

ISBN: (纸本)9781479999255

Over the past years, frameworks such as MapReduce and Spark have been introduced to ease the task of developing big data programs and applications. However, the jobs in these frameworks are roughly defined and packaged as executable jars without any functionality being exposed or described. This means that deployed jobs are not natively composable and reusable for subsequent development. Besides, it also hampers the ability for applying optimizations on the data flow of job sequences and pipelines. In this paper, we present the Hierarchically Distributed data Matrix (HDM) which is a functional, strongly-typed data representation for writing composable big data applications. Along with HDM, a runtime framework is provided to support the execution of HDM applications on distributed infrastructures. Based on the functional data dependency graph of HDM, multiple optimizations are applied to improve the performance of executing HDM jobs. The experimental results show that our optimizations can achieve improvements of between 10% to 60% of the Job-Completion-Time for different types of operation sequences when compared with the current state of art, Apache Spark.

关键词： big data processing parallel programming functional programming distributed systems system architecture

来源：评论

学校读者我要写书评

暂无评论

Analysis of Effective E-Commerce Coordination big data processing Strategies under Infinite Deep Neural Network Topology

Analysis of Effective E-Commerce Coordination Big Data Proce...

引用

Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS)

作者： Zhang, Ailing Yao, Fuxiao Shandong Vocat & Tech Univ Int Studies Rizhao 276800 Shandong Peoples R China

ISBN: (纸本)9781728176499

This paper researches and analyses the effective e-commerce coordination big data processing strategy through the infinite-depth neural network topology. This paper firstly proposes a neural network training model Neural Network-Storm (NN-S) based on Storm streaming distributed architecture, which decomposes the neural network training task into multiple computing units by data-parallel method, the parameters are updated synchronously after the training of a single batch of data is completed. In the Storm architecture, a Zookeeper network is used for multi-server distributed deployment. The training results show that the NN-S model can significantly improve the training speed of neural networks. At the same time, the NN-S architecture can quickly recover from node failures and network resource scheduling abnormalities with strong robustness. In this paper, we investigate the streaming-based distributed neural network training and design a Storm-based distributed neural network training model and optimized training algorithms, which are of reference significance for distributed neural network training.

关键词： infinite depth neural network topology effective e-conunerce coordination big data processing strategy analysis

来源：评论

学校读者我要写书评

暂无评论

PrEstoCloud: Proactive Cloud Resources Management at the Edge for Efficient Real-Time big data processing 7

PrEstoCloud: Proactive Cloud Resources Management at the Edg...

引用

7th International Conference on Cloud Computing and Services Science (CLOSER)

作者： Verginadis, Yiannis Alshabani, Iyad Mentzas, Gregoris Stojanovic, Nenad Natl Tech Univ Athens Inst Commun & Comp Syst Athens Greece Activeeon Valbonne France Nissatech Innovat Ctr Nish Serbia

ISBN: (纸本)9789897582431

Among the greatest challenges of cloud computing is to automatically and efficiently exploit infrastructural resources in a way that minimises cloud fees without compromising the performance of resource demanding cloud applications. In this aspect the consideration of using processing nodes at the edge of the network, increases considerably the complexity of these challenges. PrEstoCloud idea encapsulates a dynamic, distributed, self-adaptive and proactively configurable architecture for processing big data streams. In particular, PrEstoCloud aims to combine real-time big data, mobile processing and cloud computing research in a unique way that entails proactiveness of cloud resources use and extension of the fog computing paradigm to the extreme edge of the network. The envisioned PrEstoCloud solution is driven by the microservices paradigm and has been structured across five different conceptual layers: i) Meta-management;ii) Control;iii) Cloud infrastructure;iv) Cloud/Edge communication and v) Devices, layers.

关键词： big data processing Dynamic Resources Management Edge Computing

来源：评论

学校读者我要写书评

暂无评论

Intelligent big data processing for Wind Farm Monitoring and Analysis Based on Cloud-Technologies and Digital Twins A quantitative approach 3

Intelligent Big Data Processing for Wind Farm Monitoring and...

引用

3rd IEEE International Conference on Cloud Computing and big data Analysis (ICCCBDA)

作者： Pargmann, Hergen Euhausen, Doerthe Faber, Robin Jade Univ Appl Sci Wilhelmshaven Friedrich Paffrath Str 101 D-26389 Wilhelmshaven Germany ZEIT GmbH & Co KG D-26127 Oldenburg Germany Carl von Ossietzky Univ Oldenburg Ammerlander Heerstr 114-118 D-26129 Oldenburg Germany

ISBN: (纸本)9781538643013

Monitoring and analysis processes of big data are normally bound either to a technical or business context and even further separated into continuous or discrete resolution. In our approach a sound semantical integration of these different information types is presented and applied to a concrete use-case, the global monitoring and analysis of wind farms. Furthermore, a new user-interface style based on augmented reality is presented allowing - even to a novice - an intuitive and quick understanding of complex analysis results.

关键词： big data processing augmented reality digital twin cloud computing internet of things real-time processing micro services wind farms smart control station

来源：评论

学校读者我要写书评

暂无评论

Organization of Meta-knowledge in the Form of 23-bit Templates for big data processing 5

Organization of Meta-knowledge in the Form of 23-bit Templat...

引用

5th International Conference on Computing for Geospatial Research and Application (COM.Geo)

作者： Bari, Nima Liao, Duoduo Berkovich, Simon George Washington Univ Dept Comp Sci Washington DC 20052 USA Comp Geospatial Res Inst Reston VA USA

ISBN: (纸本)9781479943210

We are living and facing an unprecedented growth of available large-scale structured and unstructured data both. From different broad range of online websites and applications, data is being collected at significantly exceptional rate. For instance, as described in [11]" Facebook reports about 6 billion new photos every month and 72 hours of video are uploaded to YouTube every minute". Researchers and developers are faced with this large amount of data that needs to be processed, analyzed, and clustered. Analysis of big data essentially drives every aspect of our daily life, including and not limited to, retail services, mobile services, financial services, manufacturing, and life sciences. The existing and conventional data processing techniques and clustering algorithms were not initially designed to handle this large amount of data and we face challenges to analyze big data. This research paper attempts to enhance these existing clustering algorithms to process big data. This research introduces an unprecedented big data processing technique using a 23-bit question meta-knowledge template for big data clustering in a linear time complexity O(n).

关键词： 23bits big data processing Bounded Rationality Clustering Golay Code Ontology Engineering Stream processing meta-knowledge Template big data Golay codes STREAMS(PROCESS) Cluster Analysis complexity classes Clustering algorithms unstructured data

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：