Static web page

SPHINX: a framework for creating personal, site-specific Web crawlers

Robert C. Miller, Krishna Bharat

Citations: 151 • 1998

SiteHelper: a localized agent that helps incremental exploration of the World Wide Web

Daniel Siaw Weng Ngu, Xindong Wu

Citations: 67 • 1997

LOGML: Log Markup Language for Web Usage Mining

John R. Punin, Mukkai S. Krishnamoorthy, Mohammed J. Zaki

Citations: 51 • 2002

The Ethicality of Web Crawlers

Yang Sun, C. Lee Giles

Citations: 39 • 2010

Who and what links to the Internet Archive

Yasmin AlNoamany, Ahmed AlSum, Michele C. Weigle, Michael L. Nelson

Citations: 32 • 2014

Information Monitoring on the Web: A Scalable Solution

LING LIU, Wei Tang, David Buttler, Calton Pu

Citations: 29 • 2002

Novel pre-processing technique for web log mining by removing global noise and web robots

P. Nithya, P. Sumathi

Citations: 26 • 2012

Analysis of Aggregated Bot and Human Traffic on E-Commerce Site

Grażyna Suchacka

Citations: 23 • 2014

A dynamic URL assignment method for parallel web crawler

Francesco Ragni, C. Martines

Citations: 22 • 2010

Why web sites are lost (and how they're sometimes found)

Frank McCown, Catherine C. Marshall, Michael L. Nelson

Citations: 20 • 2009

Characterizing Crawler Behavior from Web Server Access Logs

Marios D. Dikaiakos, Athena Stassopoulou, Loizos Papageorgiou

Citations: 20 • 2003

How to Set Up and Maintain a Web Site

Lincoln Stein

Citations: 17 • 1996

Dominos: A New Web Crawler's Design

Chabane Djeraba

Citations: 16 • 2004

Detection and confirmation of web robot requests for cleaning the voluminous web log data

Tanvir Habib Sardar, Zahid Ansari

Citations: 16 • 2014

A Web Crawler Detection Algorithm Based on Web Page Member List

Weigang Guo, Yong Zhong, Jianqin Xie

Citations: 14 • 2012

A Novel Technique for Web Log mining with Better Data Cleaning and Transaction Identification

Vellingiri

Citations: 13 • 2011

Indexing and Abstracting on the World Wide Web: An Examination of Six Web Databases

Scott Nicholson

Citations: 13 • 1997

An effective approach to enhancing a focused crawler using Google

Jae-Gil Lee, Donghwan Bae, Sansung Kim, Jungeun Kim, Mun Yong Yi

Citations: 13 • 2019

Applying Web Crawler Technologies for Compiling Parallel Corpora as one Stage of Natural Language Processing

Nilufar Abdurakhmonova, Ismailov Alisher, Guli Toirova

Citations: 10 • 2022

A study of different web-crawler behaviour

Alexander Menshchikov, Antonina Komarova, Yurij Gatchin, A.G. Кorobeynikov, Nina Tishukova

Citations: 9 • 2017