GPT has demonstrated impressive capabilities in executing various natural language processing (NLP) and reasoning tasks, showcasing its potential for deductive coding in social annotations. This research explored the ...
详细信息
ISBN:
(纸本)9798400716188
GPT has demonstrated impressive capabilities in executing various natural language processing (NLP) and reasoning tasks, showcasing its potential for deductive coding in social annotations. This research explored the effectiveness of prompt engineering and fine-tuning approaches of GPT for deductive coding of contextdependent and context-independent dimensions. coding contextdependent dimensions (i.e., Theorizing, Integration, Reflection) requires a contextualized understanding that connects the target comment with reading materials and previous comments, whereas coding context-independent dimensions (i.e., Appraisal, Questioning, Social, Curiosity, Surprise) relies more on the comment itself. Utilizing strategies such as prompt decomposition, multi-prompt learning, and a codebook-centered approach, we found that prompt engineering can achieve fair to substantial agreement with expertlabeled data across various coding dimensions. These results affirm GPT ' s potential for effective application in real-world coding tasks. Compared to context-independent coding, context-dependent dimensions had lower agreement with expert-labeled data. To enhance accuracy, GPT models were fine-tuned using 102 pieces of expert-labeled data, with an additional 102 cases used for validation. The fine-tuned models demonstrated substantial agreement with ground truth in context-independent dimensions and elevated the inter-rater reliability of context-dependent categories to moderate levels. This approach represents a promising path for significantly reducing human labor and time, especially with large unstructured datasets, without sacrificing the accuracy and reliability of deductive coding tasks in social annotation. The study marks a step toward optimizing and streamlining coding processes in social annotation. Our findings suggest the promise of using GPT to analyze qualitative data and provide detailed, immediate feedback for students to elicit deepening inquiries.
Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While rece...
详细信息
ISBN:
(纸本)9798400701078
Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined code-books to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.
Thematic analysis (TA), as a qualitative analytic method, is widely used in health care, psychology, and beyond. However, scant details are often given to demonstrate the process of data analysis, especially in the fi...
详细信息
Thematic analysis (TA), as a qualitative analytic method, is widely used in health care, psychology, and beyond. However, scant details are often given to demonstrate the process of data analysis, especially in the field of education. This article describes how a hybrid approach of TA was applied to interpret multiple data sources in a practitioner inquiry. Particular attention is given to the inductive and deductive coding and theme development process of TA. Underpinned by the constructivist epistemology, codes were driven by both data per se and theories, through a "bottom-up" and "top-down" approach to identify themes. A detailed example of six steps of data analysis is presented, which evidences the systematic analysis of raw data from observation and research journals, students' focus groups, and a classroom teacher's semistructured interviews. This example demonstrates how classroom practice was unpacked and how insiders' insights were interpreted through the theoretical lens while also allowing the participants to express themselves. By providing step-by-step guidelines in data coding and identification of themes, this article contributes to informing qualitative researchers, especially teacher-researchers who undertake their research in the classroom setting.
暂无评论