检索结果-内蒙古大学图书馆

AN OPTIMAL SCHEDULED LEARNING RATE FOR A RANDOMIZED KACZMARZ ALGORITHM

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Marshall, Nicholas F. Mickelin, Oscar Department of Mathematics Oregon State University United States Program in Applied and Computational Mathematics Princeton University United States

We study how the learning rate affects the performance of a relaxed randomized Kaczmarz algorithm for solving Ax ≈ b + Ε, where Ax = b is a consistent linear system and Ε has independent mean zero random entries. We derive a learning rate schedule which optimizes a bound on the expected error that is sharp in certain cases;in contrast to the exponential convergence of the standard randomized Kaczmarz algorithm, our optimized bound involves the reciprocal of the Lambert-W function of an exponential. Copyright © 2022, The Authors. All rights reserved.

关键词： Linear systems

A Priori Estimates of the Population Risk for Residual Networks

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Weinan, E. Ma, Chao Wang, Qingcan Department of Mathematics Princeton University Program in Applied and Computational Mathematics Princeton University Beijing Institute of Big Data Research

Optimal a priori estimates are derived for the population risk, also known as the generalization error, of a regularized residual network model. An important part of the regularized model is the usage of a new path norm, called the weighted path norm, as the regularization term. The weighted path norm treats the skip connections and the nonlinearities differently so that paths with more nonlinearities are regularized by larger weights. The error estimates are a priori in the sense that the estimates depend only on the target function, not on the parameters obtained in the training process. The estimates are optimal, in a high dimensional setting, in the sense that both the bound for the approximation and estimation errors are comparable to the Monte Carlo error rates. A crucial step in the proof is to establish an optimal bound for the Rademacher complexity of the residual networks. Comparisons are made with existing norm-based generalization error bounds. Copyright © 2019, The Authors. All rights reserved.

关键词： Error analysis

Ordered and disordered stealthy hyperuniform point patterns across spatial dimensions

学校读者我要写书评

暂无评论

Physical Review Research 2024年第3期6卷 033260页

作者： Peter K. Morse Paul J. Steinhardt Salvatore Torquato Department of Chemistry Department of Physics Princeton Institute of Materials Princeton Center for Theoretical Science Program in Applied and Computational Mathematics

In previous work [Phys. Rev. X 5, 021020 (2015)] it was shown that stealthy hyperuniform systems can be regarded as hard spheres in Fourier space in the sense that the structure factor is exactly zero in a spherical region around the origin in analogy with the pair-correlation function of real-space hard spheres. While this earlier work focused on spatial dimensions d=1–4, here we extend the analysis to higher dimensions in order to make connections to high-dimensional sphere packings and the mean-field theory of glasses. We exploit this correspondence to confirm that the densest Fourier-space hard-sphere system is that of a Bravais lattice in contrast to real-space hard spheres, whose densest configuration is conjectured to be disordered. In passing, we give a concise form for the position of the first Bragg peak. We also extend the virial series previously suggested for disordered stealthy hyperuniform systems to higher dimensions in order to predict spatial decorrelation as a function of dimension. This prediction is then borne out by numerical simulations of disordered stealthy hyperuniform ground states in dimensions d=2–8, which have only recently been made possible due to a highly parallelized algorithm.

关键词： Classical statistical mechanics Disordered systems

A Comparative Analysis of Optimization and Generalization Properties of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Weinan, E. Ma, Chao Wu, Lei Department of Mathematics Princeton University Program in Applied and Computational Mathematics Princeton University Beijing Institute of Big Data Research

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the over-parametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to that of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error is established for target functions in the appropriate reproducing kernel Hilbert *** Codes 41A99, 49M99 Copyright © 2019, The Authors. All rights reserved.

关键词： Gradient methods

Machine learning from a continuous viewpoint

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Weinan, E. Ma, Chao Wu, Lei Department of Mathematics Princeton University Program in Applied and Computational Mathematics Princeton University Beijing Institute of Big Data Research

We present a continuous formulation of machine learning, as a problem in the calculus of variations and differential-integral equations, very much in the spirit of classical numerical analysis and statistical physics. We demonstrate that conventional machine learning models and algorithms, such as the random feature model, the shallow neural network model and the residual neural network model, can all be recovered as particular discretizations of different continuous formulations. We also present examples of new models, such as the ow-based random feature model, and new algorithms, such as the smoothed particle method and spectral method, that arise naturally from this continuous formulation. We discuss how the issues of generalization error and implicit regularization can be studied under this framework. Copyright © 2019, The Authors. All rights reserved.

关键词： Machine learning

Analysis of the gradient descent algorithm for a deep neural network model with skip-connections

学校读者我要写书评

暂无评论

arXiv 2019年

作者： Weinan, E. Ma, Chao Wang, Qingcan Wu, Lei Department of Mathematics Princeton University Program in Applied and Computational Mathematics Princeton University Beijing Institute of Big Data Research

The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path are also established. As a consequence, it is shown that when the target function is in the reproducing kernel Hilbert space (RKHS) with a kernel defined by the initialization, there exist generalizable earlystopping solutions along the GD path. In addition, it is also shown that the GD path is uniformly close to the functions given by the related random feature model. Consequently, in this \implicit regularization" setting, the deep neural network model deteriorates to a random feature model. Our results hold for neural networks of any width larger than the input dimension. Copyright © 2019, The Authors. All rights reserved.

关键词： Gradient methods

Inertial Hegselmann-Krause systems

学校读者我要写书评

暂无评论

Inertial Hegselmann-Krause systems

American Control Conference (ACC)

作者： Bernard Chazelle Chu Wang Department of Computer Science Princeton University Program in Applied and Computational Mathematics Princeton University

ISBN: (纸本)9781467386838

We derive an energy bound for inertial Hegselmann-Krause (HK) systems, which we define as a variant of the classic HK model in which the agents can change their weights arbitrarily at each step. We use the bound to prove the convergence of HK systems with closed-minded agents, which settles a conjecture of long standing. This paper also introduces anchored HK systems and show their equivalence to the symmetric heterogeneous model.

关键词： Convergence Communication networks Kinetic theory Protocols Nickel Mobile communication Turning

Strong and weak divergence in finite time of Euler's method for stochastic differential equations with non-globally Lipschitz continuous coefficients

学校读者我要写书评

暂无评论

Strong and weak divergence in finite time of Euler's method ...

作者： Hutzenthaler, Martin Jentzen, Arnulf Kloeden, Peter E. LMU Biozentrum Department Biologie II University of Munich 82152 Planegg-Martinsried Germany Program in Applied and Computational Mathematics Princeton University Princeton NJ 08544-1000 United States Institute for Mathematics Goethe University Frankfurt am Main 60054 Frankfurt am Main Germany

The stochastic Euler scheme is known to converge to the exact solution of a stochastic differential equation (SDE) with globally Lipschitz continuous drift and diffusion coefficients. Recent results extend this convergence to coefficients that grow, at most, linearly. For superlinearly growing coefficients, finite-time convergence in the strong mean-square sense remains. In this article, we answer this question to the negative and prove, for a large class of SDEs with non-globally Lipschitz continuous coefficients, that Euler's approximation converges neither in the strong mean-square sense nor in the numerically weak sense to the exact solution at a finite time point. Even worse, the difference of the exact solution and of the numerical approximation at a finite time point diverges to infinity in the strong mean-square sense and in the numerically weak sense. © 2010 The Royal Society.

关键词： Stochastic systems

Representation formulas and pointwise properties for Barron functions

学校读者我要写书评

暂无评论

arXiv 2020年

作者： Weinan, E. Wojtowytsch, Stephan Department of Mathematics Program in Applied and Computational Mathematics Princeton University PrincetonNJ08544 United States Princeton University Program in Applied and Computational Mathematics 205 Fine Hall - Washington Road PrincetonNJ08544 United States

We study the natural function space for infinitely wide two-layer neural networks with ReLU activation (Barron space) and establish different representation formulae. In two cases, we describe the space explicitly up to isomorphism. Using a convenient representation, we study the pointwise properties of two-layer networks and show that functions whose singular set is fractal or curved (for example distance functions from smooth submanifolds) cannot be represented by infinitely wide two-layer networks with finite path-norm. We use this structure theorem to show that the only C1-diffeomorphisms which Barron space are affine. Furthermore, we show that every Barron function can be decomposed as the sum of a bounded and a positively one-homogeneous function and that there exist Barron functions which decay rapidly at infinity and are globally Lebesgue-integrable. This result suggests that two-layer neural networks may be able to approximate a greater variety of functions than commonly *** Codes 68T07, 46E15, 26B35, 26B40 Copyright © 2020, The Authors. All rights reserved.

关键词： Network layers