Using Metadata to Enhance a Web Information Gathering System.

Abstract

With the web at lose to a billion pages and growing at an exponential rate, we are fa ed with the issue of rating pages in terms of quality and trust. In this situation, what other pages say about a web page an be as important as what the page says about itself. The umulative knowledge of these types of re ommendations (or the la k thereof) an be obje tive enough to help a user or robot program to deide whether or not to pursue a web do ument. In addition, these annotations or metadata an be used by a web robot program to derive summary information about web do uments that are written in a language that the robot does not understand. We use this idea to drive a web information gathering system that forms the ore of a topi -spe i sear h engine. In this paper, we des ribe how our system uses annotations about the hyperlinks ontained in web pages to guide itself to rawl the web. It sifts through useful information related to a parti ular topi to eliminate the traversal of links that may not be of interest. Thus, the guided rawling system stays fo used on the target topi . It builds a ri h repository of link information that in ludes annotations. This repository is used to build quality metadata, whi h ultimately serves a sear h engine.