Ruffle&Riley: Towards the Automated Induction of Conversational Tutoring Systems

作     者：Schmucker, Robin Xia, Meng Azaria, Amos Mitchell, Tom 

作者机构：Machine Learning Department Carnegie Mellon University United States Human-Computer Interaction Institute Carnegie Mellon University United States Department of Computer Science Ariel University 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2023年

核心收录：

主　　题：Natural language processing systems 

摘      要：Conversational tutoring systems (CTSs) offer learning experiences driven by natural language interaction. They are known to promote high levels of cognitive engagement and benefit learning outcomes, particularly in reasoning tasks. Nonetheless, the time and cost required to author CTS content is a major obstacle to widespread adoption. In this paper, we introduce a novel type of CTS that leverages the recent advances in large language models (LLMs) in two ways: First, the system induces a tutoring script automatically from a lesson text. Second, the system automates the script orchestration via two LLM-based agents (Ruffle&Riley) with the roles of a student and a professor in a learning-by-teaching format. The system allows a free-form conversation that follows the ITS-typical inner and outer loop structure. In an initial between-subject online user study (N = 100) comparing Ruffle&Riley to simpler QA chatbots and reading activity, we found no significant differences in post-test scores. Nonetheless, in the learning experience survey, Ruffle&Riley users expressed higher ratings of understanding and remembering and further perceived the offered support as more helpful and the conversation as coherent. Our study provides insights for a new generation of scalable CTS technologies. Copyright © 2023, The Authors. All rights reserved.