Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks

作     者：Sato, Naoki Izumi, Koshiro Iiduka, Hideaki 

作者机构：Computer Science Course Graduate School of Science and Technology Meiji University Kanagawa214-8571 Japan Visual Solutions Department 1 Technology Development Division SECOM Co.Ltd. Tokyo181-8528 Japan Department of Computer Science Meiji University Kanagawa214-8571 Japan 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2024年

核心收录：

主　　题：Generative adversarial networks 

摘      要：A scaled conjugate gradient method that accelerates existing adaptive methods utilizing stochastic gradients is proposed for solving nonconvex optimization problems with deep neural networks. It is shown theoretically that, whether with constant or diminishing learning rates, the proposed method can obtain a stationary point of the problem. Additionally, its rate of convergence with diminishing learning rates is verified to be superior to that of the conjugate gradient method. The proposed method is shown to minimize training loss functions faster than the existing adaptive methods in practical applications of image and text classification. Furthermore, in the training of generative adversarial networks, one version of the proposed method achieved the lowest Fréchet inception distance score among those of the adaptive methods. © 2024, CC BY.