The Artificial Social Agent Questionnaire (ASAQ) — Development and evaluation of a validated instrument for capturing human interaction experiences with artificial social agents

Abstract

Validating claims and replicating findings on the impact of artificial social agents (ASA), such as virtual agents, conversational agents, and social robots, requires a standardised measurement instrument that researchers can employ in different settings and for various agents. Such an instrument would allow researchers to evaluate their agents and establish insights beyond their specific study context. Therefore, we present the long and short versions of the ASA questionnaire (ASAQ) for evaluating human-ASA interaction on 19 constructs, such as the agent’s believability, sociability, and coherence. It has been developed by an international workgroup with more than 100 ASA-researchers over multiple years who identified community-relevant constructs and associated questionnaire items and examined the questionnaire’s reliability, validity, and interpretability. The result is a questionnaire that can capture more than 80% of the constructs that studies in the intelligent virtual agent community investigate, with acceptable levels of reliability, content validity, construct validity, and cross-validity. We suggest that ASA-researchers use the ASAQ short version to report their agent’s psychographic information and the ASAQ long version to analyse any constructs in-depth that are specifically relevant to their agent or study. Finally, this paper gives instructions for practical use, such as sample size estimations, and how to interpret and present results. • The Artificial-Social-Agent Questionnaire (ASAQ) is a validated measure used to evaluate the human experience of interacting with an artificial social agent (ASA). • There are two versions of the ASAQ. The short version is used to report the ASAs psychographic information, while the long version allows for in-depth analysis of constructs relevant to a specific ASA. • More than 100 ASA researchers were involved in the development of ASAQ. • The 19 ASAQ constructs capture over 80% of the constructs investigated in the Intelligent Virtual Agent community between 2013-2018. • The ASAQ has demonstrated acceptable levels of reliability, content validity, construct validity, and cross-validity. • We present the ASAQ representative set 2024 of 29 agents. • Instructions for the practical use of ASAQ are given, including guidance on choosing the ASAQ version, estimating sample sizes, and interpreting and presenting the results (e.g., ASAQ chart).