A topic-specific Web robot model based on restless bandits

Abstract

Constructing and maintaining topic-specific Web indexes is modeled by a restless-bandits generalization and resolved by a reinforcement-learning algorithm. The authors outline the potential role of topic-specific robots in distributed search engine design, and they model the complex problem of automatically constructing and maintaining topic-specific Web indexes. Experimental results establish the viability of a topic-specific Web robot design based on the restless bandit model. The results indicate that the proposed algorithm is a good foundation on which to build a complete solution.