Imitation learning through games: theory, implementation and evaluation

摘要

Despite a history of games-based research, academia has generally regarded
\ncommercial games as a distraction from the serious business of AI, rather than as an
\nopportunity to leverage this existing domain to the advancement of our knowledge.
\nSimilarly, the computer game industry still relies on techniques that were developed
\nseveral decades ago, and has shown little interest in adopting more progressive
\nacademic approaches. In recent times, however, these attitudes have begun to change;
\nunder- and post-graduate games development courses are increasingly common,
\nwhile the industry itself is slowly but surely beginning to recognise the potential
\noffered by modern machine-learning approaches, though games which actually
\nimplement said approaches on more than a token scale remain scarce.
\nOne area which has not yet received much attention from either academia or industry
\nis imitation learning, which seeks to expedite the learning process by exploiting data
\nharvested from demonstrations of a given task. While substantial work has been done
\nin developing imitation techniques for humanoid robot movement, there has been
\nvery little exploration of the challenges posed by interactive computer games. Given
\nthat such games generally encode reasoning and decision-making behaviours which
\nare inherently more complex and potentially more interesting than limb motion data,
\nthat they often provide inbuilt facilities for recording human play, that the generation
\nand collection of training samples is therefore far easier than in robotics, and that
\nmany games have vast pre-existing libraries of these recorded demonstrations, it is
\nfair to say that computer games represent an extremely fertile domain for imitation
\nlearning research.
\nIn this thesis, we argue in favour of using modern, commercial computer games to
\nstudy, model and reproduce humanlike behaviour. We provide an overview of the
\nbiological and robotic imitation literature as well as the current status of game AI, highlighting techniques which may be adapted for the purposes of game-based
\nimitation. We then proceed to describe our contributions to the field of imitation
\nlearning itself, which encompass three distinct categories: theory, implementation
\nand evaluation.
\nWe first describe the development of a fully-featured Java API - the Quake2 Agent
\nSimulation Environment (QASE) - designed to facilitate both research and education
\nin imitation and general machine-learning, using the game Quake 2 as a testbed. We
\noutline our motivation for developing QASE, discussing the shortcomings of existing
\nAPIs and the steps which we have taken to circumvent them. We describe QASE’s
\nnetwork layer, which acts as an interface between the local AI routines and the
\nQuake 2 server on which the game environment is maintained, before detailing the
\nAPI’s agent architecture, which includes an interface to the MatLab programming
\nenvironment and the ability to parse and analyse full recordings of game sessions.
\nWe conclude the chapter with a discussion of QASE’s adoption by numerous
\nuniversities as both an undergraduate teaching tool and research platform.
\nWe then proceed to describe the various imitative mechanisms which we have
\ndeveloped using QASE and its MatLab integration facilities. We first outline a
\nbehaviour model based on a well-known psychological model of human planning.
\nDrawing upon previous research, we also identify a set of believability criteria -
\nelements of agent behaviour which are of particular importance in determining the
\n“humanness” of its in-game appearance. We then detail a reinforcement-learning
\napproach to imitating the human player’s navigation of his environment, centred
\nupon his pursuit of items as strategic goals