How Public is the Web?: Robots, Access, and Scholarly Communication

Abstract

This paper examines the use of "Robot Exclusion Protocol" to restrict the access of search engine robots to 10 major American university websites belonging to institutions recently named among "AmericaÕs Most Wired" universities (Gan, 1997). An analysis of web site searching and interviews with web server administrators at these sites shows that the decision to use this procedure is largely technical and is typically made by the web server administrator. The implications of this decision for openness in scholarly communication and for the future of academic, university-based web publishing are discussed.