Monte-Carlo Planning for Socially Aligned Agents using Bayesian Affect Control Theory

Abstract

Affect Control Theory (ACT) is a mathematically well-defined model that makes ac-curate predictions about the affective content of human action. The affective predictions, which are derived from statistics about human actions and identities in real and labora-tory environments, are shared normative behaviours that are believed to lead to solutions to everyday cooperative problems. A probabilistic and decision-theoretic generalisation of ACT, called BayesAct, allows the principles of ACT to be used for human-interactive agents by defining a utility function and a probabilistic version of the ACT dynamical model of affect. Planning in BayesAct, which we address in this paper, then allows one to go beyond the affective norm, and leads to the emergence of more complex interac-tions between “cognitive ” reasoning and “affective ” reasoning, such as deception lead-ing to manipulation and altercasting. As BayesAct is a large hybrid (continuous-discrete) state/action/observation partially observable Markov decision process (POMDP), in this paper we propose a continuous variant of a successful Monte-Carlo tree search planner (POMCP), which performs dynamic discretisation of the action and observation spaces while planning. We demonstrate our variant POMCP-C in simulation on (i) a two-agent coordination problem that involves manipulation through affective interaction, and (ii) an affectively-aware assistive health-care device. In addition, we show that our solver can be used in non-affective domains, by demonstrating it on a continuous robot navi-gation problem from the literature and achieving over 50 % increase in average reward compared to traditional solvers. 1