Cognitive Shifts in Bilingual Speakers Affect Speech Interactions with Artificial Agents

Abstract

AbstractA major research question in psycholinguistics relates to the phenomenon of linguistic relativity, which contends that the language one speaks influences how one thinks. Of particular interest is whether bilingual speakers shift cognitive paradigms when speaking different languages. Here, we conducted a human-agent interaction (HAI) study using a bilingual virtual avatar capable of autonomous speech during cooperative gameplay in two languages (Korean and English). We ran 40 participants, including 20 monolingual speakers (10 Korean, 10 English) and 20 Korean/English bilingual speakers, engaging the avatar during 30-minute game sessions. Comparison of speech patterns showed that bilingual speakers exhibited notable "cognitive shifts" in both languages while interacting with the avatar, which were markedly different from their monolingual counterparts. Interestingly, the virtual avatar's own speech behavior also significantly changed during interaction with bilingual speakers, despite identical programming. As evidenced here, such cognitive shifts appear to impact the way humans interact with artificial agents.Keywords: Human-robot interactionsocial cognitionbilingualismvirtual avatarlinguistic relativitycross-cultural robotics AcknowledgmentsWe would also like to thank our various collaborators, including Jaeyoung Suh, Jihong Jeong, and Seeun Lee (Hanyang University), for their assistance in this work.Data availability statementThe data used in this study included audiovisual recordings of participants during the experiment. However, it may be made available in de-identifiable form upon reasonable request.Disclosure statementThe authors report there are no competing interests to declare.Additional informationFundingThis work was supported by a grant from the National Research Foundation of Korea (NRF) under grant number: 2021R1G1A1003801.Notes on contributorsCasey C. BennettCasey C. Bennett is an Associate Professor in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, as well as DePaul University in Chicago IL (USA). He specializes in artificial intelligence and robotics in healthcare, including use of data science and machine learning to create better human-robot interaction.Say Young KimSay Young Kim is an Associate Professor in the Department English Language & Literature at Hanyang University in Seoul, Korea. He specializes in the study of language and bilingualism. He completed his PhD at Maryland University in the US.Benjamin WeissBenjamin Weiss is a researcher at Quality and Usability Lab at Technische Universität Berlin in Germany and has also worked at Hanyang University in Seoul Korea previously. He specializes in natural language processing and speech systems. He received his PhD from Humboldt University in Berlin Germany.Young-Ho BaeYoung-Ho Bae is a graduate student in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL). His research focuses on natural language processing and the application of data science to human-robot interaction.Jun Hyung YoonJun Hyung Yoon is a graduate student in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL). His research focuses on natural language processing and the application of data science to human-robot interaction.Yejin ChaeYejin Chae is a undergraduate research assistant in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL).Eunseo YoonEunseo Yoon is a undergraduate research assistant in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL).Uijae RyuUijae Ryu is a undergraduate research assistant in the Department of Intelligence Computing at Hanyang University in Seoul, Korea, working in the Data Science Innovation Lab (DSIL).Hansae