URL diving

Abstract

Perhaps the most important difference among search sites is how they are structured, both internally and externally. Sites such as Yahoo include a highly structured system of categories to help users drill down to the site they want. Others, such as AltaVista, accept only word-based searches with no categorical structure or browsing at all. There are also those that offer combinations of searching and browsing, with some variations to set them apart, such as ratings for some of the sites in their database, enabling searches among only prequalified sites. Finally, there are the so-called "metasearch" sites, which have no indexes of their own, but rather query, refilter, and compile results based on searches among multiple search engines. Perhaps the most important thing to understand is how search sites build and maintain their database indexes. There are two primary methods in use: registration and robot crawlers. Both have their own strengths and weaknesses, and consequently a place in every researcher's toolbox.