Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning

作     者：Tian, Jiawu Xu, Liwei Zhang, Xiaowei Li, Yongqi 

作者机构：School of Mathematical Sciences University of Electronic Science and Technology of China China 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Optimization algorithms 

摘      要：Training deep neural networks is a challenging task. In order to speed up training and enhance the performance of deep neural networks, we rectify the vanilla conjugate gradient as conjugate-gradient-like and incorporate it into the generic Adam, and thus propose a new optimization algorithm named CG-like-Adam for deep learning. Specifically, both the first-order and the second-order moment estimation of generic Adam are replaced by the conjugate-gradient-like. Convergence analysis handles the cases where the exponential moving average coefficient of the first-order moment estimation is constant and the first-order moment estimation is unbiased. Numerical experiments show the superiority of the proposed algorithm based on the CIFAR10/100 dataset. Copyright © 2024, The Authors. All rights reserved.