Why? Why not? When? Visual Explanations of Agent Behavior in\n Reinforcement Learning

摘要

Reinforcement learning (RL) is used in many domains, including autonomous\ndriving, robotics, stock trading, and video games. Unfortunately, the black box\nnature of RL agents, combined with legal and ethical considerations, makes it\nincreasingly important that humans (including those are who not experts in RL)\nunderstand the reasoning behind the actions taken by an RL agent, particularly\nin safety-critical domains. To help address this challenge, we introduce\nPolicyExplainer, a visual analytics interface which lets the user directly\nquery an autonomous agent. PolicyExplainer visualizes the states, policy, and\nexpected future rewards for an agent, and supports asking and answering\nquestions such as: Why take this action? Why not take this other action? When\nis this action taken? PolicyExplainer is designed based upon a domain analysis\nwith RL researchers, and is evaluated via qualitative and quantitative\nassessments on a trio of domains: taxi navigation, a stack bot domain, and drug\nrecommendation for HIV patients. We find that PolicyExplainer promotes trust\nand understanding of agent decisions better than a state-of-the-art text-based\nexplanation approach. Interviews with domain practitioners provide further\nvalidation for PolicyExplainer as applied to safety-critical domains. Our\nresults help demonstrate how visualization-based approaches can be leveraged to\ndecode the behavior of autonomous RL agents, particularly for RL non-experts.\n