A Limitation of Gradient Descent Learning

作     者：Sum, John Leung, Chi-Sing Ho, Kevin 

作者机构：Natl Chung Hsing Univ Inst Technol Management Taichung 40227 Taiwan City Univ Hong Kong Dept Elect Engn Hong Kong Peoples R China Providence Univ Dept Comp Sci & Commun Engn Taichung 43301 Taiwan 

出 版 物：《IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS》 (IEEE Trans. Neural Networks Learn. Sys.)

年 卷 期：2020年第31卷第6期

页      面：2227-2232页

核心收录：

学科分类：0808[工学-电气工程] 08[工学] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：Taiwan Ministry of Science and Technology (MOST) [105-2221-E-005-065-MY2] City University of Hong Kong 

主　　题：Additive weight noise gradient descent algorithms MNIST multiplicative weight noise 

摘      要：Over decades, gradient descent has been applied to develop learning algorithm to train a neural network (NN). In this brief, a limitation of applying such algorithm to train an NN with persistent weight noise is revealed. Let V(w) be the performance measure of an ideal NN. V(w) is applied to develop the gradient descent learning (GDL). With weight noise, the desired performance measure (denoted as J(w)) is E[ V(w)|w], where w is the noisy weight vector. Applying GDL to train an NN with weight noise, the actual learning objective is clearly not V(w) but another scalar function L(w). For decades, there is a misconception that L(w) = J(w), and hence, the actual model attained by the GDL is the desired model. However, we show that it might not: 1) with persistent additive weight noise, the actual model attained is the desired model as L(w) = J(w);and 2) with persistent multiplicative weight noise, the actual model attained is unlikely the desired model as L(w) = J(w). Accordingly, the properties of the models attained as compared with the desired models are analyzed and the learning curves are sketched. Simulation results on 1) a simple regression problem and 2) the MNIST handwritten digit recognition are presented to support our claims.