How the Internet was indexed

Abstract

The scope and characteristics of what may be considered the first three generations of automated Internet indexing systems are identified and described as to their methods of compiling their datasets, their search interfaces and their associated etymological metaphors and mythologies. These three are suggested to be: firstly, the archie system for single keyword and regular expression searches of the file lists of anonymous ftp sites; secondly, the veronica system for Boolean keyword in title searches of the world's gopher servers; thirdly, a range of software techniques known as robots and search engines, which compile searchable data bases of information accessible via the World Wide Web, such as the currently popular Lycos project at Carnegie Mellon University. The present dominance of World Wide Web client software as the preferred interface to Internet information has led to provision of methods of also using the first two systems by this single interface, and these are also noted.