检索结果-内蒙古大学图书馆

overparameterization of Deep ResNet: Zero Loss and Mean-field Analysis

JOURNAL OF MACHINE LEARNING RESEARCH 2022年第1期23卷 1-65页

作者： Ding, Zhiyan Chen, Shi Li, Qin Wright, Stephen J. Univ Wisconsin Madison Dept Math Madison WI 53706 USA Univ Wisconsin Madison Dept Comp Sci Madison WI 53706 USA

Finding parameters in a deep neural network (NN) that fit training data is a nonconvex optimization problem, but a basic first-order optimization method (gradient descent) finds a global optimizer with perfect fit (zero-loss) in many practical situations. We examine this phenomenon for the case of Residual Neural Networks (ResNet) with smooth activation functions in a limiting regime in which both the number of layers (depth) and the number of weights in each layer (width) go to infinity. First, we use a mean-field-limit argument to prove that the gradient descent for parameter training becomes a gradient flow for a probability distribution that is characterized by a partial differential equation (PDE) in the large-NN limit. Next, we show that under certain assumptions, the solution to the PDE converges in the training time to a zero-loss solution. Together, these results suggest that the training of the ResNet gives a near-zero loss if the ResNet is large enough. We give estimates of the depth and width needed to reduce the loss below a given threshold, with high probability.

关键词： Residual neural network overparameterization mean-field analysis zero loss gradient flow

来源：评论

学校读者我要写书评

暂无评论

overparameterization and Generalization Error: Weighted Trigonometric

引用

SIAM JOURNAL ON MATHEMATICS OF DATA SCIENCE 2022年第2期4卷 885-908页

作者： Xie, Yuege Chou, Hung-Hsu Rauhut, Holger Ward, Rachel Univ Texas Austin Oden Inst Austin TX 78712 USA Rhein Westfal TH Aachen Chair Math Informat Proc D-52056 Aachen N Rhine Westpha Germany Univ Texas Austin Math Dept Austin TX 78712 USA

Motivated by surprisingly good generalization properties of learned deep neural networks in overparameterized scenarios and by the related double descent phenomenon, this paper analyzes the relation between smoothness and low generalization error in an overparameterized linear learning problem. We study a random Fourier series model, where the task is to estimate the unknown Fourier coefficients from equidistant samples. We derive exact expressions for the generalization error of both plain and weighted least squares estimators. We show precisely how a bias toward smooth interpolants, in the form of weighted trigonometric interpolation, can lead to smaller generalization error in the overparameterized regime compared to the underparameterized regime. This provides insight into the power of overparameterization, which is common in modern machine learning.

关键词： overparameterization generalization error weighted optimization smoothness

来源：评论

学校读者我要写书评

暂无评论

overparameterization in Neural Networks: From Application to Theory

Overparameterization in Neural Networks: From Application to...

引用

作者： Zhang, Kaiqi University of California Santa Barbara

学位级别：Ph.D., Doctor of Philosophy

Neural networks are rapidly increasing in size, leading to a common occurrence of overparameterization in deep learning. This presents challenges in both the theory and application of deep learning. From a theoretical standpoint, it remains an open question as to why neural networks generalize well despite overparameterization. From an application perspective, overparameterization leads to significant computation and storage costs, which limits the practical application of deep neural networks. This thesis presents our attempt to address both issues. In terms of application, we propose training a low-rank tensorized neural network to compress the model and reduce the computation cost during both training and inference. We also apply Bayesian methods to evaluate the uncertainty of this model. In terms of theory, we apply a new method — neural tangent kernel (NTK) — to study the training dynamics of an infinitely wide neural network. We compare the eigenvalues of the NTK of a vanilla neural network with that of a binary weight neural network, and show that the latter decays faster. This explains why binary weight neural networks have lower generalization gap empirically. We also examine the effect of weight decay on a neural network, and demonstrate that it induces sparsity in both a parallel neural network and a ResNet, thus prove that neural networks are locally adaptive, which is not present in any linear method, including kernels. For the problems discussed above, we present both theoretical analyses of our method or statement, and numerical experiments to validate our conclusions.

关键词： Neural network overparameterization Computation cost Deep learning Neural tangent kernel

来源：评论

学校读者我要写书评

暂无评论

DETECTION OF overparameterization AND OVERFITTING IN AN AUTOMATIC CALIBRATION OF SWAT

引用

TRANSACTIONS OF THE ASABE 2010年第5期53卷 1487-1499页

作者： Whittaker, G. Confesor, R., Jr. Di Luzio, M. Arnold, J. G. USDA ARS Natl Forage Seed Prod Res Ctr Corvallis OR 97331 USA Heidelberg Univ Natl Ctr Water Qual Res Tiffin OH USA Texas A&M Univ Syst Blackland Res Ctr Temple TX USA USDA ARS Grassland Soil & Water Res Lab Temple TX 76502 USA

Distributed hydrologic models based on small-scale physical processes tend to have a large number of parameters to represent spatial heterogeneity. This characteristic requires the use of a large number of parameters in model calibration. It is a common view that calibration with a large number parameters produces overparameterization and overfitting. Recent work using prior information, spatial information, and constraints on parameters for regularization of the calibration problem has improved model predictions using a few dozen parameters. We demonstrate that the Soil and Water Assessment Tool (SWAT) and the information associated with a SWAT watershed setup provide a regularized problem with many of recently published regularization techniques already utilized in SWAT Our hypothesis is that the Soil and Water Assessment Tool (SWAT) regularizes the inverse problem so that a stable solution can be obtained for calibration of SWAT using a very large number of parameters, where very large means up to 10,000 calibration parameters. In this study, a two-objective calibration genetic algorithm based on a non-dominated sorting genetic algorithm (NSGA-II) was used to calibrate the Blue River basin in Oklahoma. We introduce the use of intermediate solutions found by the genetic algorithm to test identification of calibration parameters and diagnose model overfitting. Defining identification as the capability of a model to constrain the estimation of parameters, we introduced a method for statistically testing for changes from the initial uniform distribution of each parameter. We found that all 4,198 parameters used to calculate the Blue River SWAT model were identified. Diagnostic comparisons of goodness-of-fit measures for the calibration and validation periods provided strong evidence that the model was not overfitted.

关键词： Automatic calibration Distributed hydrologic model NSGA-II Overfitting overparameterization Regularization

来源：评论

学校读者我要写书评

暂无评论

A Statistical Variable Selection Solution for RFM Ill-Posedness and overparameterization Problems

引用

IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 2018年第7期56卷 3990-4001页

作者： Moghaddam, Sayyed Hamed Alizadeh Mokhtarzade, Mehdi Naeini, Amin Alizadeh Amiri-Simkooei, AliReza Khajeh Nasir Toosi Univ Technol Fac Geodesy & Geomat Engn Tehran *** Iran Univ Isfahan Fac Civil Engn & Transportat Dept Geomat Engn Esfahan *** Iran

Parameters of a rational function model (RFM), known as rational polynomial coefficients, are commonly redundant and highly correlated, leading to the problems of overparameterization and ill-posedness, respectively. In this paper, an innovative two-stage statistical method, called an uncorrelated and statistically significant RFM (USS-RFM), is presented to deal directly with these two problems. In the first stage, the proposed method employs a novel correlation analysis, which aims to exclude highly correlated coefficients. In the second stage, a new iterative significance test is applied to detect and remove unnecessary coefficients from the RFM. The proposed method is implemented on eight real data sets captured by Cartosat-1, GeoEye-1, Pleiades, Spot-3, and WorldView-3 platforms. The results are evaluated in terms of the positioning accuracy, model degrees of freedom, processing time, and figure condition analysis. Experimental results prove the efficiency of the proposed method, showing that it could achieve subpixel accuracy even for cases with five ground control points. The proposed USS-RFM is compared to an l(1)-norm regularization (L1R) technique and a particle swarm optimization (PSO) algorithm in the terrain-dependent case of the RFM. The results demonstrate the superiority of the USS-RFM, which performs better than the alternative methods in terms of positioning accuracy by more than 50% on average. Moreover, the RFMs resulted from the USS-RFM demonstrate to have higher degrees of freedom and, as a result, higher level of reliability. From the perspective of processing time, USS-RFM and L1R are similar while both are much faster than PSO.

关键词： Ill-posedness overparameterization rational function model (RFM) statistical solution

来源：评论

学校读者我要写书评

暂无评论

Toward Moderate overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks

IEEE JOURNAL ON SELECTED AREAS IN INFORMATION THEORY

引用

IEEE JOURNAL ON SELECTED AREAS IN INFORMATION THEORY 2020年第1期1卷 84-105页

作者： Oymak, Samet Soltanolkotabi, Mahdi Univ Calif Riverside Dept Elect & Comp Engn Riverside CA 92521 USA Univ Southern Calif Ming Hsieh Dept Elect Engn Los Angeles CA 90089 USA

Many modern neural network architectures are trained in an overparameterized regime where the parameters of the model exceed the size of the training dataset. Sufficiently overparameterized neural network architectures in principle have the capacity to fit any set of labels including random noise. However, given the highly nonconvex nature of the training landscape it is not clear what level and kind of overparameterization is required for first order methods to converge to a global optima that perfectly interpolate any labels. A number of recent theoretical works have shown that for very wide neural networks where the number of hidden units is polynomially large in the size of the training data gradient descent starting from a random initialization does indeed converge to a global optima. However, in practice much more moderate levels of overparameterization seems to be sufficient and in many cases overparameterized models seem to perfectly interpolate the training data as soon as the number of parameters exceed the size of the training data by a constant factor. Thus there is a huge gap between the existing theoretical literature and practical experiments. In this paper we take a step towards closing this gap. Focusing on shallow neural nets and smooth activations, we show that (stochastic) gradient descent when initialized at random converges at a geometric rate to a nearby global optima as soon as the square-root of the number of network parameters exceeds the size of the training data. Our results also benefit from a fast convergence rate and continue to hold for non-differentiable activations such as Rectified Linear Units (ReLUs).

关键词： Neural network training overparameterization nonconvex optimization random matrix theory

来源：评论

学校读者我要写书评

暂无评论

More is less: inducing sparsity via overparameterization

引用

INFORMATION AND INFERENCE-A JOURNAL OF THE IMA 2023年第3期12卷 iaad012-iaad012页

作者： Chou, Hung-Hsu Maly, Johannes Rauhut, Holger Ludwig Maximilian Univ Munich Math Inst Theresienstr 39 D-80333 Munich Germany Rhein Westfal TH Aachen Chair Math Informat Proc Pontdriesch 12-14 D-52062 Aachen Germany

In deep learning, it is common to overparameterize neural networks, that is, to use more parameters than training samples. Quite surprisingly training the neural network via (stochastic) gradient descent leads to models that generalize very well, while classical statistics would suggest overfitting. In order to gain understanding of this implicit bias phenomenon, we study the special case of sparse recovery (compressed sensing) which is of interest on its own. More precisely, in order to reconstruct a vector from underdetermined linear measurements, we introduce a corresponding overparameterized square loss functional, where the vector to be reconstructed is deeply factorized into several vectors. We show that, if there exists an exact solution, vanilla gradient flow for the overparameterized loss functional converges to a good approximation of the solution of minimal l(1)-norm. The latter is well-known to promote sparse solutions. As a by-product, our results significantly improve the sample complexity for compressed sensing via gradient flow/descent on overparameterized models derived in previous works. The theory accurately predicts the recovery rate in numerical experiments. Our proof relies on analyzing a certain Bregman divergence of the flow. This bypasses the obstacles caused by non-convexity and should be of independent interest.

关键词： overparameterization l(1)-minimization gradient flow gradient descent compressed sensing implicit bias deep factorization Bregman divergence

来源：评论

学校读者我要写书评

暂无评论

Investigating overparameterization for Non-Negative Matrix Factorization in Collaborative Filtering 21

Investigating Overparameterization for Non-Negative Matrix F...

引用

15th ACM Conference on Recommender Systems (RECSYS)

作者： Kawakami, Yuhi Sugiyama, Mahito Natl Inst Informat Tokyo Japan Grad Univ Adv Studies SOKENDAI Tokyo Japan

ISBN: (纸本)9781450384582

overparameterization is one of the key techniques in modern machine learning, where a model with the higher complexity can generalize better on test data against the common knowledge of the bias-variance trade-off in classical statistical learning theory. In this paper, we empirically investigate the effect of overparameterization for matrix factorization-based models in collaborative filtering. Surprisingly, we firstly show that the performance of overparameterized non-negative matrix factorization (NMF) on test data gets better than that of the underparameterized NMF, which is commonly used to date, and is even competitive with the state-of-the-art collaborative filtering techniques. Moreover, we also show that the double descent phenomenon occurs when we increase the number of parameters of the NMF, where the test error decreases, increases, and decreases again as the model complexity grows, which has been recently reported in various machine learning methods such as deep learning models and kernel methods.

关键词： non-negative matrix factorization overparameterization recommender system collaborative filtering

来源：评论

学校读者我要写书评

暂无评论

overparameterization of deep ResNet: zero loss and mean-field analysis

The Journal of Machine Learning Research

引用

The Journal of Machine Learning Research 2022年第1期23卷 2282-2346页

作者： Zhiyan Ding Shi Chen Qin Li Stephen J. Wright Mathematics Department University of Wisconsin-Madison Madison WI Department of Computer Sciences University of Wisconsin-Madison Madison WI

关键词： residual neural network overparameterization mean-field analysis zero loss gradient flow

来源：评论

学校读者我要写书评

暂无评论

EM algorithm using overparameterization for the multivariate skew-normal distribution

引用

ECONOMETRICS AND STATISTICS 2021年 19卷 151-168页

作者： Abe, Toshihiro Fujisawa, Hironori Kawashima, Takayuki Ley, Christophe Hosei Univ Fac Econ 4342 Aihara Machida Tokyo 1940298 Japan Inst Stat Math 10-3 Midori Cho Tachikawa Tokyo 1908562 Japan Tokyo Inst Technol Dept Math & Comp Sci Meguro Ku 2-12-1 Ookayama Tokyo 1528550 Japan RIKEN Ctr Adv Intelligence Project Chuo Ku 1-4-1 Nihonbashi Tokyo 1030027 Japan Univ Ghent Dept Appl Math Comp Sci & Stat Krijgslaan 281S9Campus Sterre B-9000 Ghent Belgium

A stochastic representation with a latent variable often enables us to make an EM algorithm to obtain the maximum likelihood estimate. The skew-normal distribution has such a simple stochastic representation with a latent variable, and consequently one expects to have a convenient EM algorithm. However, even for the univariate skew-normal distribution, existing EM algorithms constructed using a stochastic representation require a solution of a complicated estimating equation for the skewness parameter, making it difficult to extend such an idea to the multivariate skew-normal distribution. A stochastic representation with overparameterization is proposed, which has not been discussed yet. The approach allows the construction of an efficient EM algorithm in a closed form, which can be extended to a mixture of multivariate skew-normal distributions. The proposed EM algorithm can be regarded as an accelerated version with momentum (which is known as an acceleration technique of the algorithm in optimization) of a recently proposed EM algorithm. The novel EM algorithm is applied to real data and compared with the command msn. mle from the R package sn. (C) 2021 EcoSta Econometrics and Statistics. Published by Elsevier B.V. All rights reserved.

关键词： EM algorithm Maximum likelihood estimate Multivariate skew-normal distribution Momentum overparameterization

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：