检索结果-内蒙古大学图书馆

detection of web site visitors based on fuzzy rough sets

SOFT COMPUTING 2018年第7期22卷 2175-2188页

作者： Hamidzadeh, Javad Zabihimayvan, Mahdieh Sadeghi, Reza Sadjad Univ Technol Fac Comp Engn & Informat Technol Mashhad Iran Imam Reza Int Univ Dept Comp Engn Mashhad Iran

Despite emerging of web 2.0 applications and increasing requirements to well-behaved web robots, malicious ones can reveal irreparable risks for web sites. Regardless of behavior of web robots, they may occupy bandwidth and reduce performance of web servers. In spite of many prestigious researches trying to characterize web visitors and classify them, there is a lack of concentration on feature selection to dynamically choose attributes used to describe web sessions. On the other hand, depending on an accurate clustering technique, which can deal with huge number of samples in a reasonable amount of time, is practically important. Therefore, in this paper, a new algorithm, fuzzy rough set-web robot detection (FRS-WRD), is proposed based on fuzzy rough set theory to better characterize and cluster web visitors of three real web sites. External evaluations show that in contrast to state-of-the-art algorithms, FRS-WRD achieves better results in terms of G-mean 95%, Jaccard 88%, entropy 0.36, and finally, purity 96%. Moreover, according to confusion matrixes, it can better detect malicious web visitors.

关键词： web robot detection Fuzzy rough set Clustering

来源：评论

学校读者我要写书评

暂无评论

Classification of web robots: An empirical study based on over one billion requests

引用

COMPUTERS & SECURITY 2009年第8期28卷 795-802页

作者： Lee, Junsup Cha, Sunydeok Lee, Dongkun Lee, Hyungkyu Korea Univ Dept Comp Sci & Engn Seoul 136701 South Korea ETRI Knowledge E Learning Res Team Taejon 305700 South Korea Korea Adv Inst Sci & Technol Dept EECS Div Comp Sci Taejon 305701 South Korea

Many studies on detection and classification of web robots have focused their attention mostly on text crawlers, and empirical experiments used relatively small data collected at universities. In this paper, we analyzed more than one billion requests to *** in 24 h. web logs were made anonymous to eliminate potential privacy concerns while preserving essential characteristics (e.g., frequency, queries, etc). We have developed an effective characterization metrics, based on workload characteristics and resource types, in detecting and classifying various web robots including text crawlers, link checkers, and icon crawlers. As expected, web robot behavior was clearly different from that of typical interactive users, and different types of web robots also exhibited different characteristics. However, comparison of the similar type of web robots, text crawlers in particular, revealed different characteristics, thereby enabling characterization with reasonably high confidence level. we divided various feature metrics into five groups, and effectiveness of each group in classification is shown in polar diagram in the decreasing order of effectiveness in the clockwise direction. One can use the findings to classify likely identify of unknown web robots, and organizations can develop appropriate measures to deal with them. Our analysis is based on recent web log data collected at one of the best known site which offers truly global service. Crown Copyright (C) 2009 Published by Elsevier Ltd. All rights reserved.

关键词： web robot classification web robot detection web robot characterization web security web usage mining

来源：评论

学校读者我要写书评

暂无评论

Discovery of web robot sessions based on their navigational patterns

引用

DATA MINING AND KNOWLEDGE DISCOVERY 2002年第1期6卷 9-35页

作者： Tan, PN Kumar, V Univ Minnesota Dept Comp Sci Minneapolis MN 55455 USA

web robots are software programs that automatically traverse the hyperlink structure of the World Wide web in order to locate and retrieve information. There are many reasons why it is important to identify visits by web robots and distinguish them from other users. First of all, e-commerce retailers are particularly concerned about the unauthorized deployment of robots for gathering business intelligence at their web sites. In addition, web robots tend to consume considerable network bandwidth at the expense of other users. Sessions due to web robots also make it more difficult to perform clickstream analysis effectively on the web data. Conventional techniques for detecting web robots are often based on identifying the IP address and user agent of the web clients. While these techniques are applicable to many well-known robots, they may not be sufficient to detect camouflaged and previously unknown robots. In this paper, we propose an alternative approach that uses the navigational patterns in the click-stream data to determine if it is due to a robot. Experimental results on our Computer Science department web server logs show that highly accurate classification models can be built using this approach. We also show that these models are able to discover many camouflaged and previously unidentified robots.

关键词： web usage mining web robot detection classification data mining

来源：评论

学校读者我要写书评

暂无评论

robots Still Outnumber Humans in web Archives, But Less Than Before 26th

Robots Still Outnumber Humans in Web Archives, But Less Than...

引用

26th International Conference on Theory and Practice of Digital Libraries (TPDL)

作者： Jayanetti, Himarsha R. Garg, Kritika Alam, Sawood Nelson, Michael L. Weigle, Michele C. Old Dominion Univ Norfolk VA 23529 USA Internet Arch Wayback Machine San Francisco CA 94118 USA

ISBN: (纸本)9783031168024;9783031168017

To identify robots and humans and analyze their respective access patterns, we used the Internet Archive's (IA) Wayback Machine access logs from 2012 and 2019, as well as ***'s (Portuguese web Archive) access logs from 2019. We identified user sessions in the access logs and classified those sessions as human or robot based on their browsing behavior. To better understand how users navigate through the web archives, we evaluated these sessions to discover user access patterns. Based on the two archives and between the two years of IA access logs (2012 vs. 2019), we present a comparison of detected robots vs. humans and their user access patterns and temporal preferences. The total number of robots detected in IA 2012 is greater than in IA 2019 (21% more in requests and 18% more in sessions). robots account for 98% of requests (97% of sessions) in *** (2019). We found that the robots are almost entirely limited to "Dip" and "Skim" access patterns in IA 2012, but exhibit all the patterns and their combinations in IA 2019. Both humans and robots show a preference for web pages archived in the near past.

关键词： web archiving User access patterns web server logs web usage mining web robot detection

来源：评论

学校读者我要写书评

暂无评论

Access Patterns for robots and Humans in web Archives 13

Access Patterns for Robots and Humans in Web Archives

引用

13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL)

作者： AlNoamany, Yasmin Weigle, Michele C. Nelson, Michael L. Old Dominion Univ Norfolk VA 23529 USA

ISBN: (纸本)9781450320764

Although user access patterns on the live web are well-understood, there has been no corresponding study of how users, both humans and robots, access web archives. Based on samples from the Internet Archive's public Wayback Machine, we propose a set of basic usage patterns: Dip (a single access), Slide (the same page at different archive times), Dive (different pages at approximately the same archive time), and Skim (lists of what pages are archived, i.e., Time-Maps). robots are limited almost exclusively to Dips and Skims, but human accesses are more varied between all four types. robots outnumber humans 10:1 in terms of sessions, 5:4 in terms of raw HTTP accesses, and 4:1 in terms of megabytes transferred. robots almost always access Time-Maps (95% of accesses), but humans predominately access the archived web pages themselves (82% of accesses). In terms of unique archived web pages, there is no overall preference for a particular time, but the recent past (within the last year) shows significant repeat accesses.

关键词： web Archiving web Server Logs web Usage Mining User Access Patterns web robot detection

来源：评论

学校读者我要写书评

暂无评论

HoneySpam 2.0: Profiling web Spambot Behaviour

HoneySpam 2.0: Profiling Web Spambot Behaviour

引用

12th International Conference on Principles of Practice in Multi-Agent Systems (PRIMA 2009)

作者： Hayati, Pedram Chai, Kevin Potdar, Vidyasagar Talevski, Alex Curtin Univ Technol Digital Ecosyst & Business Intelligence Inst Perth WA Australia

ISBN: (纸本)9783642111600

Internet hots have been widely used for various beneficial and malicious activities on the web. In this paper we provide new insights into a new kind of hot termed as web spambot which is primarily used or spreading spam content on the web. To gain insights into web spambots, we developed a tool (HoneySpam 2.0) to track their behaviour. This paper presents two main contributions, firstly it describes the design of HoneySpam 2.0 and secondly we outline the experimental results that characterise web spambot behaviour. By profiling web spambots, we provide the foundation or identifying such hots and preventing and filtering web spam content.

关键词： honeyspam web spambot web robot detection spam detection web usage mining

来源：评论

学校读者我要写书评

暂无评论

A Novel Defense Mechanism Against web Crawlers Intrusion

A Novel Defense Mechanism Against Web Crawlers Intrusion

引用

10th International Conference on Electronics, Computer and Computation (ICECCO)

作者： Aghamohammadi, Alireza Eydgahi, Ali Eastern Michigan Univ Sch Engn Technol Ypsilanti MI 48197 USA

ISBN: (纸本)9781479933433

web robots also known as crawlers or spiders are used by search engines, hackers and spammers to gather information about web pages. Timely detection and prevention of unwanted crawlers increases privacy and security of websites. In this paper, a novel method to identify web crawlers is proposed to prevent unwanted crawler to access websites. This new method suggests Five-factor identification process to detect unwanted crawlers. This work provides the pretest and posttest results along with a systematic evaluation of web pages with the proposed identification technique versus web pages without the proposed identification process. The outputs of logistic regression analysis for both treatment and control groups are provided to evaluate hypotheses and to answer the research questions. An experiment is performed with repeated measures for two groups with each group containing the same web pages. The main goal of this work was to address the challenge of identifying and preventing unwanted web crawlers by proposing a novel defense mechanism with identification process.

关键词： web crawler detection World Wide web Privacy Security web robot detection

来源：评论

学校读者我要写书评

暂无评论

robots still outnumber humans in web archives in 2019, but less than in 2015 and 2012

引用

INTERNATIONAL JOURNAL ON DIGITAL LIBRARIES 2024年第3期25卷 537-553页

作者： Jayanetti, Himarsha R. Garg, Kritika Alam, Sawood Nelson, Michael L. Weigle, Michele C. Old Dominion Univ Dept Comp Sci Norfolk VA 23529 USA Internet Arch Wayback Machine San Francisco CA USA

The significance of the web and the crucial role of web archives in its preservation highlight the necessity of understanding how users, both human and robot, access web archive content, and how best to satisfy this disparate needs of both types of users. To identify robots and humans in web archives and analyze their respective access patterns, we used the Internet Archive's (IA) Wayback Machine access logs from 2012, 2015, and 2019, as well as ***'s (Portuguese web Archive) access logs from 2019. We identified user sessions in the access logs and classified those sessions as human or robot based on their browsing behavior. To better understand how users navigate through the web archives, we evaluated these sessions to discover user access patterns. Based on the two archives and between the three years of IA access logs (2012 vs. 2015 vs. 2019), we present a comparison of detected robots vs. humans and their user access patterns and temporal preferences. The total number of robots detected in IA 2012 (91% of requests) and IA 2015 (88% of requests) is greater than in IA 2019 (70% of requests). robots account for 98% of requests in *** (2019). We found that the robots are almost entirely limited to "Dip" and "Skim" access patterns in IA 2012 and 2015, but exhibit all the patterns and their combinations in IA 2019. Both humans and robots show a preference for web pages archived in the near past.

关键词： web archiving User access patterns web server logs web usage mining web robot detection

来源：评论

学校读者我要写书评

暂无评论

detection of Internet robots Using a Bayesian Approach 2

Detection of Internet Robots Using a Bayesian Approach

引用

IEEE 2nd International Conference on Cybernetics (CYBCONF)

作者： Suchacka, Grazyna Sobkow, Mariusz Opole Univ Fac Math Phys & Comp Sci Opole Poland

ISBN: (纸本)9781479983223

A large part of web traffic on e-commerce sites is generated not by human users but by Internet robots: search engine crawlers, shopping bots, hacking bots, etc. In practice, not all robots, especially the malicious ones, disclose their identities to a web server and thus there is a need to develop methods for their detection and identification. This paper proposes the application of a Bayesian approach to robot detection based on characteristics of user sessions. The method is applied to the web traffic from a real e-commerce site. Results show that the classification model based on the cluster analysis with the Ward's method and the weighted Euclidean metric is very effective in robot detection, even obtaining accuracy of above 90%.

关键词： Bayesian approach Bayesian statistics cluster analysis correlation analysis data mining web mining web server web traffic log file analysis e-commerce Internet robot web bot web robot detection Matlab

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：