A Quantitative Metric for Reproducibility in Human Robot Interaction Experiments

Abstract

Standardization of reporting is a common initiative across many scientific disciplines. However, in the field of Human-Robot Interaction (HRI), there is a notable issue regarding the insufficient amount of information provided in papers. Researchers in the HRI community have observed that much of the critical information required to ensure reproducible and generalizable studies is often missing from published works [1]. If studies cannot be replicated, it undermines the reliability and confidence in the knowledge produced within the field. Recognizing the importance of reproducibility, this study aims to address two key questions: (1) What are the essential characteristics that must be reported for a study to be reproducible? and (2) Is it possible to measure reproducibility? By answering these questions, we developed a quantitative metric designed to help researchers assess reproducibility. To achieve this, we identified the critical information that should be included in papers to enable the replication of experiments and developed an objective questionnaire to identify the presence of those characteristics in a paper. By quantifying the responses to these questions, we constructed a metric that assigns a Reproducibility Score to each methodology. Our results demonstrate the initial feasibility of the proposed approach, introducing a potential tool to assist researchers in assessing reproducibility.