Entropic gradient descent algorithms and wide flat minima*

作     者：Pittorino, Fabrizio Lucibello, Carlo Feinauer, Christoph Perugini, Gabriele Baldassi, Carlo Demyanenko, Elizaveta Zecchina, Riccardo 

作者机构：Bocconi Univ Inst Data Sci & Analyt AI Lab I-20136 Milan Italy Politecn Torino Dept Appl Sci & Technol I-10129 Turin Italy 

出 版 物：《JOURNAL OF STATISTICAL MECHANICS-THEORY AND EXPERIMENT》 (J. Stat. Mech. Theory Exp.)

年 卷 期：2021年第2021卷第12期

核心收录：

学科分类：07[理学] 070201[理学-理论物理] 0702[理学-物理学] 0801[工学-力学（可授工学、理学学位）] 

基　　金：European Research Council European Research Council (ERC) Funding Source: European Research Council (ERC) 

主　　题：deep learning machine learning message-passing algorithms 

摘      要：The properties of flat minima in the empirical risk landscape of neural networks have been debated for some time. Increasing evidence suggests they possess better generalization capabilities with respect to sharp ones. In this work we first discuss the relationship between alternative measures of flatness: the local entropy, which is useful for analysis and algorithm development, and the local energy, which is easier to compute and was shown empirically in extensive tests on state-of-the-art networks to be the best predictor of generalization capabilities. We show semi-analytically in simple controlled scenarios that these two measures correlate strongly with each other and with generalization. Then, we extend the analysis to the deep learning scenario by extensive numerical validations. We study two algorithms, entropy-stochastic gradient descent and replicated-stochastic gradient descent, that explicitly include the local entropy in the optimization objective. We devise a training schedule by which we consistently find flatter minima (using both flatness measures), and improve the generalization error for common architectures (e.g. ResNet, EfficientNet).