A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation

作     者：Borsci, Simone Schmettow, Martin Malizia, Alessio Chamberlain, Alan van der Velde, Frank 

作者机构：Group Faculty of Behavioural Management and Social Sciences University of Twente Enschede Netherlands Department of Surgery and Cancer Faculty of Medicine NIHR London IVD Imperial College of London London United Kingdom Computer Science Department University of Pisa Pisa Italy Molde University College Molde Norway School of Computer Science University of Nottingham Nottingham United Kingdom 

出 版 物：《Personal and Ubiquitous Computing》 (Pers. Ubiquitous Comp.)

年 卷 期：2023年第27卷第2期

页      面：317-330页

核心收录：

学科分类：1205[管理学-图书情报与档案管理] 08[工学] 080204[工学-车辆工程] 0802[工学-机械工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：We would like to thank the students Imke Silderhuis  Mirre Van Bos  Jasmin Sophie Bigga  Niklas Pollmann  and Sabin Manuela Kerwien Lopez  for their help in gathering the data. Dr. Alan Chamberlain\u2019s part in this work was supported by the Engineering and Physical Sciences Research Council [grant number EP/T51729X/1] projects RCUK Catapult Researchers in Residence award [grant number EP/ V00784X/1] UKRI Trustworthy Autonomous Systems Hub and [grant number EP/S035362/1] PETRAS 2 

主　　题：Surveys 

摘      要：The Bot Usability Scale (BUS) is a standardised tool to assess and compare the satisfaction of users after interacting with chatbots to support the development of usable conversational systems. The English version of the 15-item BUS scale (BUS-15) was the result of an exploratory factorial analysis;a confirmatory factorial analysis tests the replicability of the initial model and further explores the properties of the scale aiming to optimise this tool seeking for the stability of the original model, the potential reduction of items, and testing multiple language versions of the scale. BUS-15 and the usability metrics for user experience (UMUX-LITE), used here for convergent validity purposes, were translated from English to Spanish, German, and Dutch. A total of 1292 questionnaires were completed in multiple languages;these were collected from 209 participants interacting with an overall pool of 26 chatbots. BUS-15 was acceptably reliable;however, a shorter and more reliable solution with 11 items (BUS-11) emerged from the data. The satisfaction ratings obtained with the translated version of BUS-11 were not significantly different from the original version in English, suggesting that the BUS-11 could be used in multiple languages. The results also suggested that the age of participants seems to affect the evaluation when using the scale, with older participants significantly rating the chatbots as less satisfactory, when compared to younger participants. In line with the expectations, based on reliability, BUS-11 positively correlates with UMUX-LITE scale. The new version of the scale (BUS-11) aims to facilitate the evaluation with chatbots, and its diffusion could help practitioners to compare the performances and benchmark chatbots during the product assessment stage. This tool could be a way to harmonise and enable comparability in the field of human and conversational agent interaction. © 2022, The Author(s).