Fuzzing with Sequence Diversity Inference for Sequential Decision-making Model Testing

Abstract

Nowadays increasing AI techniques, e.g., reinforcement learning, imitation learning, etc., are applied to solve sequential decision-making problems by modeling them as Markov Decision Process (MDP), and achieve superior performance in areas, such as video games, robotics and autonomous driving etc. The reliability of such models is facing severe challenges especially in some safety-critical areas, where failures would bring intolerable disasters. Existing works testing episodic decision-making models are not workable, since they neglect the nature of sequentiality and interactivity in MDP. While other works testing sequential decision-making models are challenged by low testing efficiency because the interaction of MDP is time-consuming. In this paper, we propose an optimized fuzzing framework SeqDivFuzz which infers the sequence diversity during the MDP interaction process to effectively and efficiently test sequential decision-making models in blackbox settings. It adapts the existing fuzzing framework, including Seed Selection, Seed Mutation, Feedback Analysis and integrating a module of Diversity Inference to accelerate the fuzzing procedure. The module learns historical in-process information to check the diversity of test cases when running up to checkPoint in the course of MDP, and early terminating those non-diverse ones. We conduct experimental evaluation with four models involving three simulation environments. The results reflect that SeqDivFuzz exposes 12.3%~49.1% more crashes during a 12-hour testing procedure in four pairs of models and environments compared with the state-of-the-art fuzzing framework. The idea of in-process terminating can potentially boost other techniques for testing sequential decision-making models.