PAFT: Prompt-Agnostic Fine-Tuning

作     者：Wei, Chenxing Shu, Yao Ou, Mingwen He, Ying Tiffany Yu, Fei Richard 

作者机构：College of Computer Science and Software Engineering Shenzhen University China Tsinghua Shenzhen International Graduate School Tsinghua University China  China School of Information Technology Carleton University Canada 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2025年

核心收录：

摘      要：While Large Language Models (LLMs) adapt well to downstream tasks after fine-tuning, this adaptability often compromises prompt robustness, as even minor prompt variations can significantly degrade performance. To address this, we propose prompt-agnostic fine-tuning (PAFT), a simple yet effective approach that dynamically adjusts prompts during fine-tuning. This encourages the model to learn underlying task principles rather than overfitting to specific prompt formulations. PAFT operates in two stages: First, a diverse set of meaningful, synthetic candidate prompts is constructed. Second, during fine-tuning, prompts are randomly sampled from this set to create dynamic training inputs. Extensive experiments across diverse datasets and LLMs demonstrate that models trained with PAFT exhibit strong robustness and generalization across a wide range of prompts, including unseen ones. This enhanced robustness improves both model performance and inference speed while maintaining training efficiency. Ablation studies further confirm the effectiveness of PAFT. Copyright © 2025, The Authors. All rights reserved.