The "LLM World of Words" English free association norms generated by large language models

作     者：Abramski, Katherine Improta, Riccardo Rossetti, Giulio Stella, Massimo 

作者机构：University of Pisa Department of Computer Science Pisa Italy University of Trento Department of Psychology and Cognitive Science Trento Italy National Research Council of Italy Institute of Information Science and Technologies Pisa Italy 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Semantics 

摘      要：Free associations have been extensively used in cognitive psychology and linguistics for studying how conceptual knowledge is organized. Recently, the potential of applying a similar approach for investigating the knowledge encoded in LLMs has emerged, specifically as a method for investigating LLM biases. However, the absence of large-scale LLM-generated free association norms that are comparable with human-generated norms is an obstacle to this new research direction. To address this limitation, we create a new dataset of LLM-generated free association norms modeled after the Small World of Words (SWOW) human-generated norms consisting of approximately 12,000 cue words. We prompt three LLMs, namely Mistral, Llama3, and Haiku, with the same cues as those in the SWOW norms to generate three novel comparable datasets, the LLM World of Words (LWOW). Using both SWOW and LWOW norms, we construct cognitive network models of semantic memory that represent the conceptual knowledge possessed by humans and LLMs. We demonstrate how these datasets can be used for investigating implicit biases in humans and LLMs, such as the harmful gender stereotypes that are prevalent both in society and LLM outputs. © 2024, CC BY.