Toward Integrating ChatGPT Into Satellite Image Annotation Workflows: A Comparison of Label Quality and Costs of Human and Automated Annotators

作     者：Beck, Jacob Kemeter, Lukas Malte Duerrbeck, Konrad Abdalla, Mohamed Hesham Ibrahim Kreuter, Frauke 

作者机构：Ludwig Maximilians Univ Munchen Munich Ctr Machine Learning MCML Inst Informat D-80538 Munich Germany Fraunhofer Inst Integrated Circuits IIS Ctr Appl Res Supply Chain Serv D-90411 Nurnberg Germany 

出 版 物：《IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING》 (IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.)

年 卷 期：2025年第18卷

页      面：4366-4381页

核心收录：

学科分类：0808[工学-电气工程] 1002[医学-临床医学] 08[工学] 0705[理学-地理学] 0816[工学-测绘科学与技术] 

基　　金：Munich Center for Machine Learning (MCML) Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics - Data - Applications (ADA-Center) within the framework of BAYERN DIGITAL II [20-3410-2-9-8] 

主　　题：Annotations Satellite images Biological system modeling Costs Chatbots Training Image annotation Visualization Remote sensing Data models Automated annotations ChatGPT label quality large language models (LLMs) satellite image annotation 

摘      要：High-quality annotations are a critical success factor for machine learning (ML) applications. To achieve this, we have traditionally relied on human annotators, navigating the challenges of limited budgets and the varying task-specific expertise, costs, and availability. Since the emergence of large language models (LLMs), their popularity for generating automated annotations has grown, extending possibilities and complexity of designing an efficient annotation strategy. Increasingly, computer vision capabilities have been integrated into general-purpose LLMs like ChatGPT. This raises the question of how effectively LLMs can be used in satellite image annotation tasks and how they compare to traditional annotator types. This study presents a comprehensive investigation and comparison of various human and automated annotators for image classification. We evaluate the feasibility and economic competitiveness of using the ChatGPT4-V model for a complex land usage annotation task and compare it with alternative human annotators. A set of satellite images is annotated by a domain expert and 15 additional human and automated annotators, differing in expertise and costs. Our analyzes examine the annotation quality loss between the expert and other annotators. This comparison is conducted through, first, descriptive analyzes, second, fitting linear probability models, and third, comparing F1-scores. Ultimately, we simulate annotation strategies where samples are split according to an automatically assigned certainty score. Routing low-certainty images to human annotators can cut total annotation costs by over 50% with minimal impact on label quality. We discuss implications regarding the economic competitiveness of annotation strategies, prompt engineering, and the task-specificity of expertise.