检索结果-内蒙古大学图书馆

web robot detection based on pattern-matching technique

JOURNAL OF INFORMATION SCIENCE 2012年第2期38卷 118-126页

作者： Kwon, Shinil Kim, Young-Gab Cha, Sungdeok Korea Univ Coll Informat & Commun Seoul 136701 South Korea

In web robot detection it is important is to find features that are common characteristics of diverse robots, in order to differentiate between them and humans. Existing approaches employ fairly simple features (e.g. empty referrer field, interval between successive requests), which often fail to reflect web robots' behaviour accurately. False alarms may therefore occur unacceptably often. In this paper we propose a fresh approach that expresses the behaviour of interactive users and various web robots in terms of a sequence of request types, called request patterns. Previous proposals have primarily targeted the detection of text crawlers, but our approach works well on many other web robots, such as image crawlers, email collectors and link checkers. In empirical evaluation of more than 1 billion requests collected at ***, our approach achieved 94% accuracy in web robot detection, estimated by F-measure. A decision tree algorithm proposed by Tan and Kumar was also applied to the same data. A comparison shows that the proposed approach is more accurate, and that real-time detection of web robots is feasible.

关键词： web robot detection web robot pattern human pattern pattern analysis

来源：评论

学校读者我要写书评

暂无评论

web robot detection techniques: overview and limitations

引用

DATA MINING AND KNOWLEDGE DISCOVERY 2011年第1-2期22卷 183-210页

作者： Doran, Derek Gokhale, Swapna S. Univ Connecticut Dept Comp Sci & Engn Storrs CT 06269 USA

Most modern web robots that crawl the Internet to support value-added services and technologies possess sophisticated data collection and analysis capabilities. Some of these robots, however, may be ill-behaved or malicious, and hence, may impose a significant strain on a web server. It is thus necessary to detect web robots in order to block undesirable ones from accessing the server. Such detection is also essential to ensure that the robot traffic is considered appropriately in the performance and capacity planning of web servers. Despite a variety of web robot detection techniques, there is no consensus regarding a single technique, or even a specific "type" of technique, that performs well in practice. Therefore, to aid in the development of a practically applicable robot detection technique, this survey presents a critical analysis and comparison of the prevalent detection approaches. We propose a framework to classify the existing detection techniques into four categories based on their underlying detection philosophy. We compare the different classes to gain insights into those characteristics that make up an effective robot detection scheme. Finally, we discuss why the contemporary techniques fail to offer a general solution to the robot detection problem and propose a set of key ingredients necessary for strong web robot detection.

关键词： web Crawler web robot WWW web robot detection web User Classification

来源：评论

学校读者我要写书评

暂无评论

web robot detection techniques based on statistics of their requested URL resources

Web robot detection techniques based on statistics of their ...

引用

9th International Conference on Computer Supported Cooperative Work in Design

作者： Guo, WG Ju, SG Gu, Y Foshan Univ Informat Ctr Foshan 528000 Guangdong Peoples R China

ISBN: (纸本)1846000025

Following the widely use of search engines, the impact web robots have on the, web sites should not be ignored After analyzing the navigational patterns of web robots from web logs, two new algorithms are proposed One is based on classification and statistics of requested URL resources, which classifies the URL resources into eight types and counts the number of session of the clients and number of visiting records with same type. And another is based on web page member list, which constructs one member list for every web page and one show table for every visitor. The experiment shows that the two new algorithms can detect the unknown robots and unfriendly robots who do not obey the Standard for robot Exclusion.

关键词： search engine web robot detection content classification webpage member list web log

来源：评论

学校读者我要写书评

暂无评论

web robot detection: A Semantic Approach 30

Web Robot Detection: A Semantic Approach

引用

30th IEEE International Conference on Tools with Artificial Intelligence (ICTAI)

作者： Lagopoulos, Athanasios Tsoumakas, Grigorios Papadopoulos, Georgios Aristotle Univ Thessaloniki Sch Informat Thessaloniki Greece Atypon Syst LLC Santa Clara CA USA

ISBN: (纸本)9781538674499

web robots constitute nowadays more than half of the total web traffic. Malicious robots threaten the security, privacy and performance of the web, while non-malicious ones are involved in analytics skewing. The latter constitutes an important problem for large websites with unique content, as it can lead to false impressions about the popularity and impact of a piece of information. To deal with this problem, we present a novel web robot detection approach for content-rich websites, based on the assumption that human web users are interested in specific topics, while web robots crawl the web randomly. Our approach extends the typical representation of user sessions with a novel set of features that capture the semantics of the content of the requested resources. Empirical results on real-world data from the web portal of an academic publisher, show that the proposed semantic features lead to improved web robot detection accuracy.

关键词： web robot detection crawler semantics. content analysis supervised learning

来源：评论

学校读者我要写书评

暂无评论

FRS-SIFS: fuzzy rough set session identification and feature selection in web robot detection

引用

INTERNATIONAL JOURNAL OF MACHINE LEARNING AND CYBERNETICS 2024年第2期15卷 237-252页

作者： Hamidzadeh, Javad Rahimi, Samaneh Zarif, Mohammad Ali Sadjad Univ Fac Comp Engn & Informat Technol Mashhad Iran

Nowadays, web robots are a big part of web and useful in many cases. But, there are malicious web robots that need to be detected. web robots often conceal their navigations by sending requests with incorrect or no information. It can be quite difficult to correctly and precisely classify this kind of incomplete data, including missing values. Previous studies have used IP addresses and user agent names to overcome this challenge, but these methods are unreliable. In order to solve this challenge, this paper has presented a robust algorithm named FRS-SIFS (Fuzzy Rough Set Session Identification and Feature Selection). FRS-SIFS first identifies user sessions using fuzzy rough set clustering based on string similarity measures. It then determines important features for recognizing web users' behavioral patterns using fuzzy rough set classification. FRS-SIFS labels the sessions using a novel precise heuristic method based on four phases. Moreover, two different feature selection methods are used which include fuzzy rough set quick reduction algorithm and a novel wrapper feature selection method. Finally, the multi-objective optimization algorithm NSGA-II (non-dominated sorting genetic algorithm II) is used to select the optimal set of features. The performance of the proposed method has been evaluated on a real-world dataset by the tenfold cross-validation method. The results of the experiments have been compared with state-of-the-art methods which show the superiority of the proposed method in terms of recall, precision, and F1 measures.

关键词： web robot detection Rough set theory Fuzzy rough set theory Session identification Data classification

来源：评论

学校读者我要写书评

暂无评论

An integrated method for real time and offline web robot detection

引用

EXPERT SYSTEMS 2016年第6期33卷 592-606页

作者： Doran, Derek Gokhale, Swapna S. Wright State Univ Dept Comp Sci & Engn Kno E Sis Res Ctr Dayton OH 45435 USA Univ Connecticut Dept Comp Sci & Engn Storrs CT 06269 USA

Recent academic and industry reports confirm that web robots dominate the traffic seen by web servers across the Internet. Because web robots crawl in an unregulated fashion, they may threaten the privacy, function, performance, and security of web servers. There is therefore a growing need to be able to identify robot visitors automatically, in offline and in real time, to assess their impact and to potentially protect web servers from abusive bots. Yet contemporary detection approaches, which rely on syntactic log analysis, finding statistical variations between robot and human traffic, analytical learning techniques, or complex software modifications may not be realistic to implement or remain effective as the behavior of robots evolve over time. Instead, this paper presents a novel detection approach that relies on the differences in the resource request patterns of web robots and humans. It rationalizes why differences in resource request patterns are expected to remain intrinsic to robots and humans despite the continuous evolution of their traffic. The performance of the approach, adoptable for both offline and real time settings with a simple implementation, is demonstrated by playing back streams of actual web traffic with varying session lengths and proportions of robot requests.

关键词： intrusion detection offline detection online detection real-time detection web robot web robot detection web crawler web crawler detection

来源：评论

学校读者我要写书评

暂无评论

web-S4AE: a semi-supervised stacked sparse autoencoder model for web robot detection

引用

NEURAL COMPUTING & APPLICATIONS 2023年第24期35卷 17883-17898页

作者： Jagat, Rikhi Ram Sisodia, Dilip Singh Singh, Pradeep Natl Inst Technol Raipur Dept Comp Sci & Engn Raipur 492010 India

web robots are automated computer programs that can be exploited for benign and malicious activities such as website indexing, monitoring, or unauthorized content scraping and scalping. Several methods are available to detect automated web robots through their footprints and behaviors. Although the accuracy and efficiency of existing methods depend highly on the labeled web log data, countless web requests are generated daily with the help of web robots. Exhaustive and accurate manual labeling of reconstructed sessions is time-consuming and challenging. Further, effective detection of web robots is more challenging with unlabeled or partially labeled data. To address the aforementioned issues, we reformulated web robot detection as a semi-supervised learning problem. In this paper, we propose a deep learning-based Semi-Supervised Stacked Sparse AutoEncoder (web-S4AE) for web robot detection. The proposed model uses content-based features and features extracted from web access log data to effectively classify web robots. The experiments were conducted on publicly available web log data from a library and information portal to assess the performance of web-S4AE. The web-S4AE model was trained in two phases. The first phase;comprises training the model with unlabeled data to extract the hidden information, and in the second phase, the model is fine-tuned using labeled data. The results suggest that incorporating more unlabeled data can significantly improve the classifier's performance. The web-S4AE model's performance was also compared with other models such as the Decision Tree (DT), Random Forest (RF), eXtreme Gradient Boosting (XGBoost), and Multi-Layer Perceptron (MLP).

关键词： web robot detection Semi-supervised learning Machine learning Deep learning Deep feature extraction Stacked sparse autoencoder

来源：评论

学校读者我要写书评

暂无评论

A soft computing approach for benign and malicious web robot detection

引用

EXPERT SYSTEMS WITH APPLICATIONS 2017年 87卷 129-140页

作者： Zabihimayvan, Mandieh Sadeghi, Reza Rude, H. Nathan Doran, Derek Wright State Univ Dept Comp Sci & Engn Kno E Sis Res Ctr Dayton OH 45435 USA

The accurate detection of web robot sessions from a web server log is essential to take accurate traffic level measurements and to protect the performance and privacy of information on a web server. Moreover, the irrecoverable risks of visits from malicious robots that intentionally try to evade web server intrusion detection systems, covering-up their visits with fabricated fields in their http request packets, cannot be ignored. To separate both types of robots from humans in practice, analysts turn to heuristic methods or state-of-the-art soft computing approaches that have only been tuned to the specification of a kind of web server. Noting that the landscape of web robot agents is ever changing, and that behavioral patterns and characteristics vary across different web servers, both options are lacking. To overcome this challenge, this paper presents SMART, a soft computing system that simultaneously detects benign and malicious types of robot agents from web server logs and can automatically adapt to the session characteristics of a web server. The results of experiments over some access log file servers, each servicing different domains of the web, demonstrate outperformance of the proposed method on state-of-the-art ones for benign and malicious robot detection. (C) 2017 Elsevier Ltd. All rights reserved.

关键词： Markov clustering algorithm web robot detection web crawler Malicious web agents Fuzzy Rough Set Theory

来源：评论

学校读者我要写书评

暂无评论

A Density Based Clustering Approach for web robot detection 4

A Density Based Clustering Approach for Web Robot Detection

引用

4th International Conference on Computer and Knowledge Engineering (ICCKE)

作者： Zabihi, Mahdieh Jahan, Majid Vafaei Hamidzadeh, Javad Imam Reza Int Univ Comp Engn Mashhad Iran Islamic Azad Univ Dept Comp Engn Mashhad Iran Sadjad Univ Technol Fac Comp Engn & Informat Technol Mashhad Iran

ISBN: (纸本)9781479954865

Distinction between humans and web robots, in terms of computer network security, has led to the robot detection problem. An exact solution for this issue can preserve web sites from the intrusion of malicious robots and increase the performance of web servers by prioritizing human users. In this article, we propose a density based method called DBC_WRD (Density Based Clustering for web robot detection) to discover the traffic of web robots on two large real data sets. So, we assume the visitors as the spatial instances and introduce two new features to describe and distinguish them. These attributes are based on the behavioral patterns of web visitors and remain invariant over time. By focusing on one of the disadvantages of DBSCAN as the density based clustering algorithm used in this paper, we just utilize 4 features to reduce the dimensions. According to the supervised evaluations, DBC_WRD can have the 96% of Jaccard metric and produce two clusters which have the entropy and purity rates of 0.0215 and 0.97, respectively. Furthermore, the comparisons show that from the standpoint of clustering quality and accuracy, DBC_WRD performs better than state-of-the-art algorithms. Finally, it can be concluded that some non-malicious popular web robots, through imitating the human's behavior, make it difficult to be identified.

关键词： density based Clustering DBSCAN data mining web robot detection behavioral patterns of web visitors

来源：评论

学校读者我要写书评

暂无评论

detection and Confirmation of web robot Requests for Cleaning the Voluminous web Log Data

Detection and Confirmation of Web Robot Requests for Cleanin...

引用

International Conference on IMpact of E-Technology on US (IMPETUS)

作者： Sardar, Tanvir Habib Ansari, Zahid PA Coll Engn Dept Informat Sci & Engn Mangalore Karnataka India PA Coll Engn Dept Comp Sci & Engn Mangalore Karnataka India

ISBN: (纸本)9781479926039

web robots are software applications that run automated tasks over the internet. They traverse the hyperlink structure of the World Wide web so that they can retrieve information. There are many reasons to distinguish web robot requests and user requests. Some tasks of web robots can be harmful to the web. Firstly, web robots are employed for assemble business intelligence at e-commerce sites. In such a state of affairs, the e-commerce site may need to detect robots. Secondly, many e-commerce sites carry out web traffic scrutiny to deduce the way their customers have accessed the site. Unfortunately, such scrutiny can be erroneous by the presence of web robots. Thirdly, web robots often consume considerable network bandwidth and server resources at the expense of other users. A web log file is a web server file automatically created and maintained by a web server to check the activity performed by it. It maintains a history of page requests on its site. In this paper we have used four methods together to detect and finally confirm requests as a robot request. Experiments have been performed on the log file generated from the server of an operational web site named vtulife. com which contains data of march-2013. In our research results of web robot detection using various techniques have been compared and an integrated approach is proposed for the confirmation of the robot request.

关键词： web usage mining web robot detection web log file

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：