Optimized Gradient Clipping for Noisy Label Learning

作     者：Ye, Xichen Wu, Yifan Zhang, Weizhong Li, Xiaoqiang Chen, Yifan Jin, Cheng 

作者机构：Shanghai University China Fudan University China Hong Kong Baptist University Hong Kong Shanghai Key Laboratory of Intelligent Information Processing China Shanghai Collaborative Innovation Center of Intelligent Visual Computing China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

摘      要：Previous research has shown that constraining the gradient of loss function w.r.t. model-predicted probabilities can enhance the model robustness against noisy labels. These methods typically specify a fixed optimal threshold for gradient clipping through validation data to obtain the desired robustness against noise. However, this common practice overlooks the dynamic distribution of gradients from both clean and noisy-labeled samples at different stages of training, significantly limiting the model capability to adapt to the variable nature of gradients throughout the training process. To address this issue, we propose a simple yet effective approach called Optimized Gradient Clipping (OGC), which dynamically adjusts the clipping threshold based on the ratio of noise gradients to clean gradients after clipping, estimated by modeling the distributions of clean and noisy samples. This approach allows us to modify the clipping threshold at each training step, effectively controlling the influence of noise gradients. Additionally, we provide statistical analysis to certify the noise-tolerance ability of OGC. Our extensive experiments across various types of label noise, including symmetric, asymmetric, instance-dependent, and real-world noise, demonstrate the effectiveness of our *** Codes 68T07, 68T10 © 2024, CC BY.