Moral crumple zones: cautionary tales in human–robot interaction

Abstract

This chapter introduces the concept of a moral crumple zone through an analysis of several high-profile accidents involving automated sociotechnical systems and the media coverage that surrounded them. The analysis highlights how responsibility for an action may be misattributed to a human actor who has limited control over the behavior of an automated system. Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may simply become a component—accidentally or intentionally—that bears the brunt of the moral or legal responsibilities when the overall system malfunctions. The concept is both a challenge to and an opportunity for the design and regulation of human–robot systems. At stake is not only the misattribution of responsibility, but also the ways in which new forms of harm may develop in new complex, automated, or purported autonomous technologies.