Norm Learning with Reward Models from Instructive and Evaluative Feedback

摘要

People are increasingly interacting with artificial agents in social settings, and as these agents become more sophisticated, people will have to teach them social norms. Two prominent teaching methods include instructing the learner how to act, and giving evaluative feedback on the learner’s actions. Our empirical findings indicate that people naturally adopt both methods when teaching norms to a simulated robot, and they use the methods selectively as a function of the robot’s perceived expertise and learning progress. In our algorithmic work, we conceptualize a set of context-specific norms as a reward function and integrate learning from the two teaching methods under a single likelihood-based algorithm, which estimates a reward function that induces policies maximally likely to satisfy the teacher’s intended norms. We compare robot learning under various teacher models and demonstrate that a robot responsive to both teaching methods can learn to reach its goal and minimize norm violations in a navigation task for a grid world. We improve the robot’s learning speed and performance by enabling teachers to give feedback at an abstract level (which rooms are acceptable to navigate) rather than at a low level (how to navigate any particular room).