ConverSearch: Supporting Experts in Human Behavior Analysis of Conversational Videos with a Multimodal Scene Search Tool

作     者：Arakawa, Riku Maeda, Kiyosu Yakura, Hiromu 

作者机构：Carnegie Mellon Univ Pittsburgh PA 15213 USA Princeton Univ Princeton NJ USA Max Planck Inst Human Dev Berlin Germany 

出 版 物：《ACM TRANSACTIONS ON INTERACTIVE INTELLIGENT SYSTEMS》 (ACM Trans. Interact. Intelligent Syst.)

年 卷 期：2025年第15卷第1期

页      面：1-31页

核心收录：

学科分类：08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：JST ACT-X [JPMJAX200R] JSPS KAKENHI [JP21J20353] 

主　　题：behavior analysis scene search visual programming expert-AI collaboration 

摘      要：Multimodal scene search of conversations is essential for unlocking valuable insights into social dynamics and enhancing our communication. While experts in conversational analysis have their own knowledge and skills to find key scenes, a lack of comprehensive, user-friendly tools that streamline the processing of diverse multimodal queries impedes efficiency and objectivity. To address this gap, we developed ConverSearch, a visual-programming-based tool based on insights for effective interface and implementation design derived from a formative study with experts. The tool allows experts to integrate various machine learning algorithms to capture human behavioral cues without the need for coding. Our user study, employing the System Usability Scale (SUS) and satisfaction metrics, demonstrated high user preference, reflecting the tool s ease of use and effectiveness in supporting scene search tasks. Additionally, through a deployment trial within industrial organizations, we confirmed the tool s objectivity, reusability, and potential to enhance expert workflows. This suggests the advantages of expert-AI collaboration in domains requiring human contextual understanding and demonstrates how customizable, transparent tools yielding reusable artifacts can support expert-driven tasks in complex, multimodal environments.