检索结果-内蒙古大学图书馆

stream clustering guided supervised learning for classifying NIDS alerts

FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE 2024年 155卷 231-244页

作者： Vaarandi, Risto Guerra-Manzanares, Alejandro Tallinn Univ Technol Ctr Digital Forens & Cyber Secur Tallinn Estonia New York Univ Abu Dhabi Ctr Interacting Urban Networks Abu Dhabi U Arab Emirates

A Network Intrusion Detection System (NIDS) is a network monitoring technology for identifying cyber attacks, botnet command and control traffic, and other unwanted network activity. Unfortunately, organizational NIDS solutions can often generate tens or hundreds of thousands of alerts on a daily basis, with a significant part of them having low importance or being false positives. Therefore, high priority alerts become hard to spot, which overloads security analysts and complicates their work. The current paper addresses this problem and introduces a machine learning framework for classifying NIDS alerts with the help of stream clustering and supervised learning. We propose a stream -clustering -guided method for creating labeled NIDS alert data sets. The small data sets created using this method can be used for training high-performance supervised NIDS alert classifiers. This significantly reduces the human labeling effort and eases the application of supervised machine learning for NIDS alert classification. The proposed machine learning framework was evaluated on NIDS alerts collected over 2 months from the network of a large academic organization. The experimental results indicate that combining stream clustering and supervised learning into a NIDS alert classification framework significantly decreases the number of false positives, and thus reduces the workload of human security analysts. The framework also features low CPU time and memory consumption and can thus be run on commodity hardware. In conclusion, the proposed framework provides a cost-effective means of integrating machine learning into Security Operation Centers (SOCs). This enables the identification of critical NIDS alerts using high-performance classifiers, thereby assisting in the automation of alert handling tasks for SOC personnel. To address the lack of public data sets in the problem domain and foster further research, we publicly share the large labeled NIDS alert data set used in our exp

关键词： Network security Intrusion detection Network Intrusion Detection System IDS NIDS NIDS alert High-priority NIDS alert NIDS alert classification Supervised learning Small training data set stream clustering Data set generation Data labeling Security Operations Center SOC Workload reduction

来源：评论

学校读者我要写书评

暂无评论

Smoclust: synthetic minority oversampling based on stream clustering for evolving data streams

引用

MACHINE LEARNING 2024年第7期113卷 4671-4721页

作者： Chiu, Chun Wai Minku, Leandro L. Keele Univ Sch Comp Sci & Math Keele ST5 5BG Staffs England Univ Birmingham Sch Comp Sci Birmingham B15 2TT England

Many real-world data stream applications not only suffer from concept drift but also class imbalance. Yet, very few existing studies investigated this joint challenge. Data difficulty factors, which have been shown to be key challenges in class imbalanced data streams, are not taken into account by existing approaches when learning class imbalanced data streams. In this work, we propose a drift adaptable oversampling strategy to synthesise minority class examples based on stream clustering. The motivation is that stream clustering methods continuously update themselves to reflect the characteristics of the current underlying concept, including data difficulty factors. This nature can potentially be used to compress past information without caching data in the memory explicitly. Based on the compressed information, synthetic examples can be created within the region that recently generated new minority class examples. Experiments with artificial and real-world data streams show that the proposed approach can handle concept drift involving different minority class decomposition better than existing approaches, especially when the data stream is severely class imbalanced and presenting high proportions of safe and borderline minority class examples.

关键词： Data streams Class imbalance Concept drift stream clustering Synthetic data Data difficulty factors

来源：评论

学校读者我要写书评

暂无评论

Temporal silhouette: validation of stream clustering robust to concept drift

引用

MACHINE LEARNING 2024年第4期113卷 2067-2091页

作者： Vazquez, Felix Iglesias Zseby, Tanja TU Wien Inst Telecommun Gusshaus str 25-E389 A-1040 Vienna Austria

stream clustering is required in applications where data is generated continuously or periodically and must be processed considering its temporal nature. In the absence of a ground truth, internal validation is the only option to evaluate the quality of performances. Traditional internal validation is commonly used also in stream clustering, even in spite of the fact that it becomes inconsistent in the event of data evolution. Recent trends opt for incremental approaches, but these are closer to change detection rather than validation methods and limit themselves by imposing online validation on online analysis. In this work we study the impact of concept drift in the validation of stream clustering and propose the Temporal Silhouette index, therefore making internal validation conform to streaming data. We conduct tests with more than 200 datasets and contrast performances of four popular stream clustering algorithms with seven validation methods (three static internal, three incremental internal, one external) and the proposed index. Results show the suitability of the Temporal Silhouette index for stream clustering validation in the event of concept drift and different types of outliers. The demand for reliable unsupervised learning in applications that process data in streams is ever-increasing, and such reliability inevitably requires the use of validation. This fact highlights the significance of the novel approach proposed in this work.

关键词： stream clustering clustering validation Multivariate time series

来源：评论

学校读者我要写书评

暂无评论

Tracestream: Anomalous Service Localization based on Trace stream clustering with Online Feedback 34

TraceStream: Anomalous Service Localization based on Trace S...

引用

34th IEEE International Symposium on Software Reliability Engineering (ISSRE)

作者： Zhou, Tong Zhang, Chenxi Peng, Xin Yan, Zhenghui Li, Pairui Liang, Jianming Zheng, Haibing Zheng, Wujie Deng, Yuetang Fudan Univ Sch Comp Sci Shanghai Peoples R China Fudan Univ Shanghai Key Lab Data Sci Shanghai Peoples R China Shanghai Collaborat Innovat Ctr Intelligent Visua Shanghai Peoples R China Tencent Inc Shenzhen Peoples R China

ISBN: (纸本)9798350315943

Modern large-scale service-based systems such as microservice systems have become increasingly complex, making it hard to localize anomalous services when various issues emerge. Traces record the workflows of requests through service instances and have been widely used in anomaly detection and root cause analysis. Existing trace-based approaches widely use statistical methods or learning-based techniques to detect trace anomalies and localize anomalous services. However, these approaches often suffer from the concept drift problem, i.e., the statistical properties of traces change over time in unforeseen ways. In this paper, we propose Tracestream, an anomalous service localization approach based on trace data stream clustering. Tracestream uses data stream clustering to discover potential anomalous trace clusters in evolving trace data and uses spectrum analysis to localize anomalous services based on the clusters. Moreover, Tracestream can effectively incorporate the online feedback of operation engineers based on the trace clusters to improve the accuracy for localizing anomalous services. Our evaluation confirms that Tracestream can effectively detect anomalies and localize anomalous services in an evolving microservice system. It can effectively incorporate human feedback to further improve the performance of anomalous service localization. Moreover, Tracestream is efficient and its efficiency can be further improved by sampling a small portion of traces by cluster.

关键词： microservice anomaly detection fault localization distributed tracing trace stream clustering

来源：评论

学校读者我要写书评

暂无评论

Textual One-Pass stream clustering with Automated Distance Threshold Adaption 14th

Textual One-Pass Stream Clustering with Automated Distance T...

引用

14th Asian Conference on Intelligent Information and Database Systems (ACIIDS)

作者： Assenmacher, Dennis Trautmann, Heike Univ Munster Munster Germany GESIS Leibniz Inst Social Sci Cologne Germany

ISBN: (纸本)9783031217425;9783031217432

stream clustering is a technique capable of identifying homogeneous groups of observations that continuously arrive in a digital stream. In this work, we inherently refine a TF-IDF-based text stream clustering algorithm by the introduction of an automated distance threshold adaption technique for document insertion and cluster merging, improving the performance during distributional changes in the data stream. By conducting a thorough evaluation study, we show that our new fast approach outperforms state-of-the-art one-pass and batch-based stream clustering algorithms on various existing benchmarking datasets as well as a newly introduced dataset that poses additional challenges to the community. Moreover, we find that current evaluation approaches in the field of textual stream clustering are not adequate for a sound clustering performance assessment of evolving distributions. We thus demand the utilization of time-based evaluation.

关键词： stream clustering Text mining Concept drift

来源：评论

学校读者我要写书评

暂无评论

Towards an efficient real-time kernel function stream clustering method via shared nearest-neighbor density for the IIoT

引用

INFORMATION SCIENCES 2021年 566卷 364-378页

作者： Huang, Ruohe Xiao, Ruliang Zhu, Weifu Gong, Ping Chen, Jinhui Rida, Imad Fujian Noma Univ Coll Math & Informat Fuzhou 350117 Peoples R China Fujian Prov Digit Fujian Internet Of Things Lab E Fuzhou Peoples R China Hiroshima Univ Ctr Computat Social Sci Hiroshima Japan Univ Technol Compiegne Biomech & Bioengn Compiegne France

The rapid development of 5G communication technology will considerably help the expansion and gth Industrial Internet of things (IIoT). Indeed, both the data scale and dimension will significantly increase, leading to a challenging problem of the effective real-time stream clustering in the field of IIoT streaming mining. This paper proposes an efficient and novel real-time kernel function stream clustering method based on shared nearest neighbor density for IIoT. In the proposed method, the projection technology is used to select the dimensions of high-dimensional data, while the Euler kernel function is used as the similarity measure. Furthermore, the micro-clusters are divided by the shared nearest-neighbor density, and the outliers are relearned. The main innovation lies in using the Euler kernel function to measure the similarity, reduce the sensitivity of outliers, and use the relearning strategy to improve the clustering quality of the data stream. The theoretical analysis and experimental comparisons on the simulated data sets show that the proposed method is very effective and represents a good solution for clustering realtime data streams of IIoT. (c) 2021 Elsevier Inc. All rights reserved.

关键词： stream data mining stream clustering Shared nearest-neighbor density Euler kernel function Relearning strategy

来源：评论

学校读者我要写书评

暂无评论

A stream clustering Algorithm for Classifying Network IDS Alerts

A Stream Clustering Algorithm for Classifying Network IDS Al...

引用

IEEE International Conference on Cyber Security and Resilience (IEEE CSR)

作者： Vaarandi, Risto Tallinn Univ Technol Ctr Digital Forens & Cyber Secur Tallinn Estonia

ISBN: (纸本)9781665402859

Network IDS is a widely used security monitoring technology for detecting cyber attacks, malware activity, and other unwanted network traffic. Unfortunately, network IDSs are known to generate a large number of alerts which overwhelm the human analyst, with many alerts having low importance or being false positives. This paper addresses this issue and proposes a lightweight stream clustering algorithm for classifying IDS alerts and discovering frequent attack scenarios.

关键词： network IDS alert classification discovery of attack scenarios stream clustering network security monitoring

来源：评论

学校读者我要写书评

暂无评论

Subtstream: Online subtractive stream clustering algorithm

引用

CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE 2022年第15期34卷 e6968-e6968页

作者： Milli, Musa Bulut, Hasan Natl Def Univ Naval Acad Dept Comp Engn Istanbul Turkey Ege Univ Dept Comp Engn TR-35100 Izmir Turkey

Real-time stream data processing has gained high importance with the rapid rise of big data trends in different areas such as social media, finance, business, science, and bioinformatics. stream data can be characterized as fast, unstable, and big data sets. Due to these properties of stream data, it cannot be processed effectively with traditional algorithms. Just like stream data processing, clustering is also a difficult task. However, researchers have attempted to classify stream data by modifying traditional algorithms or designing new ones. So, in previous studies, incremental methods were used for clustering the stream data. This paper highlights the need to develop an efficient real-time clustering algorithm for data streams in the presence of concept high drift and an adaptive algorithm for different dimensions. The proposed clustering algorithm, Subtstream, combines decremental (subtractive property) and incremental (additivity property) strategies to overcome the high drift. It also introduces a new dimension-based approach to adopt the dimension change. We use three radius parameters, Predefined User Parameter, Proactive Adaptive Parameter, and Reactive Adaptive Parameter, to achieve adaptability. The proposed method, Subtstream, showed better performance on synthetic and real data sets.

关键词： adaptive parameter big data data stream online clustering stream clustering

来源：评论

学校读者我要写书评

暂无评论

A Dirichlet process biterm-based mixture model for short text stream clustering

引用

APPLIED INTELLIGENCE 2020年第5期50卷 1609-1619页

作者： Chen, Junyang Gong, Zhiguo Liu, Weiwen Univ Macau State Key Lab Internet Things Smart City Macau Peoples R China Univ Macau Dept Comp & Informat Sci Macau Peoples R China Chinese Univ Hong Kong Dept Comp Sci & Engn Hong Kong Peoples R China

Short text stream clustering has become an important problem for mining textual data in diverse social media platforms (e.g., Twitter). However, most of the existing clustering methods (e.g., LDA and PLSA) are developed based on the assumption of a static corpus of long texts, while little attention has been given to short text streams. Different from the long texts, the clustering of short texts is more challenging since their word co-occurrence pattern easily suffers from a sparsity problem. In this paper, we propose a Dirichlet process biterm-based mixture model (DP-BMM), which can deal with the topic drift problem and the sparsity problem in short text stream clustering. The major advantages of DP-BMM include (1) DP-BMM explicitly exploits the word-pairs constructed from each document to enhance the word co-occurrence pattern in short texts;(2) DP-BMM can deal with the topic drift problem of short text streams naturally. Moreover, we further propose an improved algorithm of DP-BMM with forgetting property called DP-BMM-FP, which can efficiently delete biterms of outdated documents by deleting clusters of outdated batches. To perform inference, we adopt an online Gibbs sampling method for parameter estimation. Our extensive experimental results on real-world datasets show that DP-BMM and DP-BMM-FP can achieve a better performance than the state-of-the-art methods in terms of NMI metrics.

关键词： Data mining stream clustering Topic modeling

来源：评论

学校读者我要写书评

暂无评论

RobustRepstream: Robust stream clustering using self-controlled connectivity graph

引用

INTELLIGENT DATA ANALYSIS 2020年第4期24卷 799-830页

作者： Callister, Ross Lazarescu, Mihai Pham, Duc-Son Curtin Univ Bentley WA 6102 Australia

A major challenge in stream clustering is the evolution in the statistical properties of the underlying data. As clustering is inherently unsupervised, selecting suitable parameter values is often difficult. clustering algorithms with sensitive parameters are often not robust to such changes, leading to poor clustering outputs. Algorithms using K-NN graphs face this problem, as they have a sensitive K-connectivity parameter which prohibits them from adapting to stream concept evolution. We address this by controlling the excess of the skewness of edge length distributions in the underlying K-NN graph by introducing novel skewness excess concept. We demonstrate the asymptotic linear dependency of skewness excess against the graph connectivity and propose the novel RobustRepstream algorithm, which extends the Repstream algorithm, and provides improved robustness against stream evolution. By automatically controlling the skewness excess, the user no longer needs to specify the K-connectivity parameter, and RobustRepstream can adjust the graph connectivity locally in order to achieve performance close to when the optimal K value is known. We demonstrate that RobustRepstream's skewness threshold parameter is insensitive and universal across all data sets. We comprehensively evaluate RobustRepstream on real-world benchmark data sets against previous stream clustering algorithms, and demonstrate that it provides better clustering performance.

关键词： clustering data streams data clustering data mining stream clustering

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：