Using Virtual Reality to Assess Auditory Performance

Abstract

Auditory assessments aim to quantify auditory abilities that impact real-world listening, such as replicating patient complaints of listening difficulty in specific situations or understanding how auditory systems perform in natural scenes. This goal suggests that assessments should be as realistic as possible. However, the need to quantify specific abilities requires greater experimental control than achievable in most natural settings. This fundamental trade-off between realism and control has existed since perception science began. But emerging technologies in virtual and augmented reality promise to bring the real world into the lab–or vice versa–and break this trade-off. Implemented correctly, these technologies offer new approaches to quasi-realistic auditory assessment with improved multisensory consistency, more natural tasks, and greater participant engagement that will make assessment more reliable, valid, and fun.audiology, virtual realityFigure 1: Compared with traditional laboratory-based assessments, VR-based versions even of simple auditory tasks can improve the consistency of auditory and visual cues, reduce distraction, and enhance participant engagement.Figure 2: Multisensory Identification, Segregation, and Localization (MISL) tasks leverage VR-based approaches to mimic listening in realistic complex scenes, support multidimensional tasks, and provide rich feedback akin to real-world listening.LIMITATIONS OF LAB-BASED ASSESSMENT Traditional laboratory-based testing remains the gold standard for auditory assessment in the clinic and in basic research. But several features of this approach limit the real-world validity of laboratory tasks. First, these tasks are rife with multisensory conflict. Booth walls and desktop displays of the testing environment are visually distracting and inconsistent with the auditory stimuli listeners are asked to judge, whereas real-world listening involves a consistent multisensory context that can reduce uncertainty and support task performance even when non-auditory cues are not explicitly informative about the task. Removing that support increases demands on participants’ memory and mental imagery—cognitive skills that can confound attempts to measure sensory abilities. Even in explicitly multisensory studies, the spatial arrangement of auditory and visual stimuli are often poorly matched: Visuals may appear on a desktop display, while sounds appear inside the head or from a distant loudspeaker. In contrast, real-world auditory and visual features are aligned in all directions and remain so as a person moves around, producing a natural experience of immersion within the sensory world. Second, laboratory-based tasks are behaviorally unrealistic, typically presenting explicit feedback about an isolated task dimension such as localization or identification. But real-world listening combines multiple task dimensions and implicit feedback. Conversation in a crowded restaurant, for example, requires simultaneous localization, segregation, and identification of target speech, along with the evaluation of facial cues that might indicate which talker asked an important question but not the content of the question itself. POTENTIAL BENEFITS OF IMMERSIVE TECHNOLOGIES Virtual reality (VR) and other immersive technologies have been gaining attention as new modes for computer-based entertainment, gaming, communication, and work. VR head-mounted displays (HMDs) present video and audio content (typically a simulated 3D environment) customized for the individual user. Low-latency tracking of the user's position in space is used to update the display in close to real time, allowing direct interaction with immersive multisensory experiences. At Boys Town National Research Hospital's Spatial Hearing Laboratory, we have been using commercial HMDs to explore the benefits of VR for auditory assessment. The focus has been on enhancing auditory assessment rather than on assessing the performance of vi