Web robot detection with semi-supervised learning method

Abstract

Web robot is an automated information gathering program that has brought a lot of problems such as information leakage, resource occupation and network security threaten. It is necessary to effectively detect and control the web access comes from web robots. Summarizes the existing categories of web robot detection method, we propose a new detection method with semi-supervised support vector machine. The experiments based on the same test data set are presented to show that the new method is superior to other robot detection methods.