Robot-Generated Databases on the World Wide Web.

Abstract

The explosive growth of World Wide Web has spurred the development of search tools to locate resources in this dynamic, diverse, and distributed environment. WWW robots, programs that attempt to gather and index WWW resources are a useful but controversial technology. This article will provide an overview of robots, robot-generated databases and comparable tools for resource discovery