On Tighter Generalization Bounds for Deep Neural Networks: CNNs, ResNets, and beyond

作     者：Li, Xingguo Lu, Junwei Wang, Zhaoran Haupt, Jarvis Zhao, Tuo 

作者机构：Computer Science Department Princeton University PrincetonNJ08540 United States Department of Operations Research and Financial Engineering Princeton University PrincetonNJ08544 United States Department of Industrial Engineering and Management Sciences Northwestern University EvanstonIL60208 United States Department of Electrical and Computer Engineering University of Minnesota MinneapolisMN55455 United States School of Industrial and Systems Engineering Georgia Institute of Technology AtlantaGA30332 United States 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2018年

核心收录：

主　　题：Deep neural networks 

摘      要：We establish a margin based data dependent generalization error bound for a general family of deep neural networks in terms of the depth and width of the networks, as well as the spectral norm of weight matrices. Through introducing a new characterization of the Lipschitz properties of neural network family, we achieve a tighter generalization error bound. Moreover, we show that the generalization bound can be further improved for bounded losses. In addition, we demonstrate that the margin scales with the product of norm, which eliminate the concern on the vacuity of the norm based bound. Aside from the general feedforward deep neural networks, our results can be applied to derive new bounds for several popular architectures, including convolutional neural networks (CNNs), residual networks (ResNets), and hyperspherical networks (SphereNets). When achieving same generalization errors with previous arts, our bounds allow for the choice of larger parameter spaces of weight matrices, inducing potentially stronger expressive ability for neural networks. Moreover, we discuss the limitation of existing generalization bounds for understanding deep neural networks with ReLU activations in classification. Copyright © 2018, The Authors. All rights reserved.