Learning Hierarchical Partially Observable Markov Decision Process Models for Robot Navigation

摘要

Abstract | We propose and investigate a general frame-work for hierarchical modeling of partially observable envi-ronments, such as oÆce buildings, using Hierarchical Hid-den Markov Models (HHMMs). Our main goal is to ex-plore hierarchical modeling as a basis for designing more eÆcient methods for model construction and useage. As a case study we focus on indoor robot navigation and show how this framework can be used to learn a hierarchy of mod-els of the environment at dierent levels of spatial abstrac-tion. We introduce the idea of model reuse that can be used to combine already learned models into a larger model. We describe an extension of the HHMM model to includes ac-tions, which we call hierarchical POMDPs, and describe a modi ed hierarchical Baum-Welch algorithm to learn these models. We train dierent families of hierarchical models for a simulated and a real world corridor environment and compare them with the standard \\\n at " representation of the same environment. We show that the hierarchical POMDP approach, combined with model reuse, allows learning hier-archical models that t the data better and train faster than at models. I.