检索结果-内蒙古大学图书馆

Optimizing Generative AI’s Accuracy and Transparency in Inductive Thematic Analysis: A Human-AI Comparison

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Nyaaba, Matthew SungEun, Min Apam, Mary Abiswin Acheampong, Kwame Owoahene Dwamena, Emmanuel Zhai, Xiaoming AI4STEM Education Center University of Georgia AthensGA United States Department of Educational Theory and Practice University of Georgia AthensGA United States Department of Elementary Middle Level Library and Technologies Education Kutztown University KutztownPA United States Department Midwifery Department College of Nursing Midwifery and Allied Health Sciences Nalerigu Ghana Department of Lifelong Education Administration and Policy University of Georgia AthensGA United States Department of Curriculum and Instruction University of Connecticut United States Department of Science Social Studies and Mathematics University of Georgia AthensGA United States

Thematic analysis is increasingly being conducted using Generative AI (GenAI), but challenges such as hallucinations, inaccuracies, and inconsistencies in various AI-driven approaches can undermine transparency and validation. To address these challenges, we employ the GPT-4 Turbo API integrated within a Stepwise Prompt-based Python Script to analyze interview data. This method is then compared with human coders' analysis. To carry out this comparison effectively, we followed a five-phase analysis and evaluation procedure. First, two independent human coders analyzed the interview transcript using the thematic analysis approaches of Clarke and Braun (2016) and Naeem et al. (2023). They followed a structured process of coding, theming, and interpreting the data. Next, an independent reviewer assessed the percentage similarity and differences between the two human coders’ analyses to evaluate their consistency. For the third phase, GenAI was used to code, theme, and interpret the same interview data. Then, an independent reviewer compared the analysis generated by GenAI to the analysis from the human coders, rating the percentage of similarity and differences between the two. Finally, we conducted a comprehensive review of the entire process to address any errors or inconsistencies that were identified along the way. The results indicated that GenAI was highly effective in generating codes, categorizing themes, and providing broader interpretations, almost matching the performance of the human coders (with nearly 100% agreement in coding and theming). However, in the interpretation phase, while both GenAI and humans conducted a theme-based interpretation, GenAI took a more generalized and conceptual approach, whereas the human coders interpretations were more specific. Additionally, when both the human and GenAI processes were mapped to Naeem et al.'s (2023) six-step thematic analysis framework, it was found that GenAI followed four of the six steps, keywording, codin

关键词： coding errors

Theoretical Analysis of Multi-coding with Non-orthogonal Signaling

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Nelson, Brian Farhang-Boroujeny, Behrouz

Even though orthogonal multi-code signaling and its derivative, simplex signaling, are well known and widely used in different communication systems, certain applications may choose to adopt non-orthogonal signaling to benefit from other advantages that such signaling methods can offer. Motivated by a class of multi-carrier spread spectrum systems, this paper presents a thorough symbol error rate analysis of the broad class of multi-code signaling methods when they make use of codes which are not necessarily orthogonal. Our analysis is also extended to the case where the code set includes the negative of each code vector, i.e., an extension to biorthogonal signaling. Moreover, it is shown that the symbol error rate results derived in this paper reduce to those available in the literature when the multi-codes are orthogonal or satisfy the correlation property of simplex multi-codes. © 2025, CC BY.

关键词： coding errors

Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science

学校读者我要写书评

暂无评论

SSRN

SSRN 2025年

作者： Brodeur, Abel Valenta, David Marcoci, Alexandru Aparicio, Juan Pablo Mikola, Derek Barbarioli, Bruno Alexander, Rohan Deer, Lachlan Stafford, Tom University of Ottawa IZA Canada University of Ottawa Canada University of Nottingham University of Cambridge United Kingdom University of Toronto Canada Tilburg University Netherlands University of Sheffield United Kingdom Department of Economics Canada The University of Western Australia Australia

This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments. We computationally reproduced quantitative results from published articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the "AI-led" approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 pp higher success rates than AI-led teams). Human teams found significantly more major errors compared to both AI-assisted teams and AI-led teams. AI-assisted teams demonstrated an advantage over more automated approaches, detecting 0.4 more major errors per team than AI-led teams, though still significantly fewer than human-only teams. Finally, both human and AI-assisted teams significantly outperformed AI-led approaches in both proposing and implementing comprehensive robustness checks. © 2025, The Authors. All rights reserved.

关键词： coding errors

Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Zhang, Kechi Li, Ge Li, Jia Dong, Yihong Jin, Zhi Ministry of Education School of Computer Science Peking University China

Code generation models have shown significant potential for automating programming tasks. However, the challenge of generating accurate and reliable code persists due to the highly complex and long-reasoning nature of the task. Even state-of-the-art models often fail in code generation due to small errors, which can drastically affect the overall functionality of code. Our study identifies that current models tend to produce errors concentrated at specific error-prone points, which significantly impacts the accuracy of the generated code. To address this issue, we introduce Focused-DPO, a framework that enhances code generation by directing preference optimization towards these critical error-prone areas. This approach builds on Direct Preference Optimization, emphasizing accuracy in parts prone to errors. Additionally, we develop a method called Error-Point Identification, which constructs a dataset that targets these problematic points without requiring costly human annotations. Our experiments on benchmarks such as HumanEval(+), MBPP(+), and LiveCodeBench demonstrate that Focused-DPO significantly improves the precision and reliability of code generation, reducing common errors and enhancing overall code quality. By focusing on error-prone points, Focused-DPO advances the accuracy and functionality of model-generated code. Copyright © 2025, The Authors. All rights reserved.

关键词： coding errors

REDO: EXECUTION-FREE RUNTIME ERROR DETECTION FOR coding AGENTS

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Shuo Kan, Andrey Callot, Laurent Bhasker, Bhavana Rashid, Muhammad Shihab Esler, Timothy B. Department of Computer Science University of Pennsylvania United States AWS AI Labs United States

As LLM-based agents exhibit exceptional capabilities in addressing complex problems, there is a growing focus on developing coding agents to tackle increasingly sophisticated tasks. Despite their promising performance, these coding agents often produce programs or modifications that contain runtime errors, which can cause code failures and are difficult for static analysis tools to detect. Enhancing the ability of coding agents to statically identify such errors could significantly improve their overall performance. In this work, we introduce Execution-free Runtime Error Detection for coding Agents (REDO), a method that integrates LLMs with static analysis tools to detect runtime errors for coding agents, without code execution. Additionally, we propose a benchmark task, SWE-Bench-Error-Detection (SWEDE), based on SWE-Bench (lite), to evaluate error detection in repository-level problems with complex external dependencies. Finally, through both quantitative and qualitative analyses across various error detection tasks, we demonstrate that REDO outperforms current state-of-the-art methods by achieving a 11.0% higher accuracy and 9.1% higher weighted F1 score;and provide insights into the advantages of incorporating LLMs for error detection. Copyright © 2024, The Authors. All rights reserved.

关键词： coding errors

On the Linearization of Optimal Rates for Independent Zero-Error Source and Channel Problems

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Charpenay, Nicolas Treust, Maël Le Roumy, Aline Univ Rennes CNRS IRMAR UMR 6625 RennesF-35000 France Univ. Rennes CNRS Inria IRISA UMR 6074 RennesF-35000 France Centre Inria l’Université de Rennes France

Zero-error coding encompasses a variety of source and channel problems where the probability of error must be exactly zero. This condition is stricter than that of the vanishing error regime, where the error probability goes to zero as the code blocklength goes to infinity. In general, zero-error coding is an open combinatorial question. We investigate two unsolved zero-error problems: the source coding problem with side information and the channel coding problem. We focus our attention on families of independent problems for which the distribution decomposes into a product of distributions, corresponding to solved zero-error problems. A crucial step is the linearization property of the optimal rate which does not always hold in the zero-error regime, unlike in the vanishing error regime. We derive a condition under which the linearization properties of the complementary graph entropy H̅ for the AND product of graph and for the disjoint union of graphs are equivalent. Then we establish the connection with a recent result obtained by Wigderson and Zuiddam and by Schrijver, for the zero-error capacity C0. As a consequence, we provide new single-letter characterizations of H̅ and C0, for example when the graph is a product of perfect graphs, which is not perfect in general, and for the class of graphs obtained by the product of a perfect graph G with the pentagon graph C5. By building on Haemers result for C0, we also show that the linearization of H̅ does not hold for the product of the Schläfli graph with its complementary graph. © 2024, CC BY.

关键词： coding errors

A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why?

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Chen, QiHong Li, Jiawei Deng, Jiecheng Yu, Jiachen Chen, Justin Tian Jin Ahmed, Iftekhar The Donald Bren School of Information and Computer Sciences University of California Irvine IrvineCA United States The University of California Riverside RiversideCA United States

Recent advancements in Large Language Models (LLMs) have led to their widespread application in automated code generation. However, these models can still generate defective code that deviates from the specification. Previous research has mainly focused on the mistakes in LLM-generated standalone functions, overlooking real-world software development situations where the successful generation of the code requires software contexts such as external dependencies. In this paper, we considered both of these code generation situations and identified a range of non-syntactic mistakes arising from LLMs’ misunderstandings of coding question specifications. Seven categories of non-syntactic mistakes were identified through extensive manual analyses, four of which were missed by previous works. To better understand these mistakes, we proposed six reasons behind these mistakes from various perspectives. Moreover, we explored the effectiveness of LLMs in detecting mistakes and their reasons. Our evaluation demonstrated that GPT-4 with the ReAct prompting technique can achieve an F1 score of up to 0.65 when identifying reasons for LLM’s mistakes, such as misleading function signatures. We believe that these findings offer valuable insights into enhancing the quality of LLM-generated code. Copyright © 2024, The Authors. All rights reserved.

关键词： coding errors

AUTODETECT: Towards a Unified Framework for Automated Weakness Detection in Large Language Models

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Cheng, Jiale Lu, Yida Gu, Xiaotao Ke, Pei Liu, Xiao Dong, Yuxiao Wang, Hongning Tang, Jie Huang, Minlie Group Tsinghua University China Zhipu AI China Tsinghua University China

Although Large Language Models (LLMs) are becoming increasingly powerful, they still exhibit significant but subtle weaknesses, such as mistakes in instruction-following or coding tasks. As these unexpected errors could lead to severe consequences in practical deployments, it is crucial to investigate the limitations within LLMs systematically. Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies, while manual inspections are costly and not scalable. In this paper, we introduce a unified framework, AUTODETECT, to automatically expose weaknesses in LLMs across various tasks. Inspired by the educational assessment process that measures students’ learning outcomes, AUTODETECT consists of three LLM-powered agents: Examiner, Questioner, and Assessor. The collaboration among these three agents is designed to realize comprehensive and in-depth weakness identification. Our framework demonstrates significant success in uncovering flaws, with an identification success rate exceeding 30% in prominent models such as ChatGPT and Claude. More importantly, these identified weaknesses can guide specific model improvements, proving more effective than untargeted data augmentation methods like Self-Instruct. Our approach has led to substantial enhancements in popular LLMs, including the Llama series and Mistral-7b, boosting their performance by over 10% across several benchmarks. Code and data are publicly available at https://***/thu-coai/AutoDetect. © 2024, CC BY.

关键词： coding errors

The Second Generalized Covering Radius of Binary Primitive Double-Error-Correcting BCH Codes

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Yohananov, Lev Schwartz, Moshe School of Electrical and Computer Engineering Ben-Gurion University of the Negev Beer Sheva8410501 Israel Department of Electrical and Computer Engineering McMaster University HamiltonONL8S 4K1 Canada

We completely determine the second covering radius for binary primitive double-error-correcting BCH codes. As part of this process, we provide a lower bound on the second covering radius for binary primitive BCH codes... 详细信息

关键词： coding errors