World Wide Web Robot for Extreme Datamining with Swiss-Tx Supercomputers

Abstract

This paper discusses the software and hardware issues of designing a highly parallel robot for extreme datamining on the Internet. As a sample application, a World Wide Web server count experiment for Switzerland and Thailand is presented. Our platform of choice is the SwissTx, a supercomputer built from commodity components that runs NT and COMPAQ Tru64 Unix. Hardware and software of this machine are discussed and benchmark results presented. They show that NT is a feasible choice even under the given extreme conditions. Using statistical modelling for optimizing the search process, the inevitable bandwidth problem is reduced to some extent to a computation problem. We suggest that our approach to Web robots is a robust bet for a multitude of future Internet applications which might lead to a large-scale and cost-efficient usage of Web robots. -- ii -- Contents 1 Introduction 1 2 The Swiss-Tx Project 2 2.1 The Swiss-T0(Dual) Machine and Future Machines . . . . . . . . . . . . . . 2...