检索结果-内蒙古大学图书馆

Depth Separation in Norm-Bounded Infinite-Width Neural Networks 37

学校读者我要写书评

暂无评论

Depth Separation in Norm-Bounded Infinite-Width Neural Netwo...

37th Annual Conference on Learning Theory, COLT 2024

作者： Parkinson, Suzanna Ongie, Greg Willett, Rebecca Shamir, Ohad Srebro, Nathan Committee on Computational and Applied Mathematics University of Chicago United States Department of Mathematical and Statistical Sciences Marquette University United States Department of Statistics Department of Computer Science University of Chicago United States Department of Computer Science and Applied Mathematics Weizmann Institute of Science Israel Toyota Technological Institute Chicago United States

We study depth separation in infinite-width neural networks, where complexity is controlled by the overall squared 2-norm of the weights (sum of squares of all weights in the network). Whereas previous depth separation results focused on separation in terms of width, such results do not give insight into whether depth determines if it is possible to learn a network that generalizes well even when the network width is unbounded. Here, we study separation in terms of the sample complexity required for learnability. Specifically, we show that there are functions that are learnable with sample complexity polynomial in the input dimension by norm-controlled depth-3 ReLU networks, yet are not learnable with sub-exponential sample complexity by norm-controlled depth-2 ReLU networks (with any value for the norm). We also show that a similar statement in the reverse direction is not possible: any function learnable with polynomial sample complexity by a norm-controlled depth-2 ReLU network with infinite width is also learnable with polynomial sample complexity by a norm-controlled depth-3 ReLU network. © 2024 S. Parkinson, G. Ongie, R. Willett, O. Shamir & N. Srebro.

关键词： Source separation

PARAMETERIZED VIETORIS-RIPS FILTRATIONS VIA COVERS

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Nelson, Bradley J. Department of Statistics Committee on Computational and Applied Mathematics University of Chicago ChicagoIL60637 United States

A challenge in computational topology is to deal with large filtered geometric complexes built from point cloud data such as Vietoris-Rips filtrations. This has led to the development of schemes for parallel computation and compression which restrict simplices to lie in open sets in a cover of the data. We extend the method of acyclic carriers to the setting of persistent homology to give detailed bounds on the relationship between Vietoris-Rips filtrations restricted to covers and the full construction. We show how these complexes can be used to study data over a base space and use our results to guide the selection of covers of data. We demonstrate these techniques on a variety of covers, and show the utility of this construction in investigating higher-order homology of a model of high-dimensional image *** Codes 55N31 (Primary), 68T09 (Secondary) © 2022, CC BY.

关键词：

Vision Transformer Pruning Via Matrix Decomposition

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Sun, Tianyi Committee on Computational and Applied Mathematics Department of Computer Science Department of Statistics University of Chicago United States

This is a further development of Vision Transformer Pruning [36] via matrix decomposition. The purpose of the Vision Transformer Pruning is to prune the dimension of the linear projection of the dataset by learning their associated importance score in order to reduce the storage, run-time memory, and computational demands. In this paper we further reduce dimension and complexity of the linear projection by implementing and comparing several matrix decomposition methods while preserving the generated important features. We end up selected the Singular Value Decomposition as the method to achieve our goal by comparing the original accuracy scores in the original Github repository and the accuracy scores of using those matrix decomposition methods, including Singular Value Decomposition, four versions of QR Decomposition, and LU factorization. © 2023, CC BY.

关键词： Singular value decomposition

Numerical Simulation on Laser Fusion in China

学校读者我要写书评

暂无评论

AIP Conference Proceedings 2009年第1期1125卷 141-153页

作者： Shaoping Zhu Wenbing Pei Yan Xu Peijun Gu Ke Lan Wenhua Ye Junfeng Wu Jinghong Li Yaoming Gao Chunyang Zheng Shuanggui Li Zeyao Mo Jun Yan Weiyan Zhang aInstitute of Applied Physics and Computational Mathematics Beijing 100088 China bNational High‐Tech Inertial Confinement Fusion Committee of China Beijing 100088 China

Numerical simulation is a powerful tool to get insight into the physics of laser fusion. Much effort has been devoted to develop the numerical simulation code series named LARED in China. The code series LARED are composed of six parts and enable us to have the simulation capability for the key processes in laser fusion. In recent years, a number of numerical simulations using LARED have been carried out and the simulation is checked by experiments done at the laser facility SG‐II and SG‐III prototype. In the present talk, some details of LARED code series will be introduced, and some simulation results, especially recent work on the opacities, will be shown.

关键词：

Optimal method for reconstructing polychromatic maps from broadband observations with an asymmetric antenna pattern

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Cantrall, Brianna Quinn, Solomon Bunn, Emory F. Department of Physics University of Richmond United States Committee on Computational and Applied Mathematics University of Chicago United States

Broadband time-ordered data obtained from telescopes with a wavelength-dependent, asymmetric beam pattern can be used to extract maps at multiple wavelengths from a single scan. This technique is especially useful when collecting data on cosmic phenomena such as the Cosmic Microwave Background (CMB) radiation, as it provides the ability to separate the CMB signal from foreground contaminants. We develop a method to determine the optimal linear combinations of wavelengths ("colors") that can be reconstructed for a given telescope design and the number of colors that are measurable with high signal-to-noise ratio. The optimal colors are found as eigenvectors of a matrix derived from the inverse noise covariance matrix. When the telescope is able to scan the sky isotropically, it is useful to transform to a spherical harmonic basis, in which this matrix has a particularly simple form. We propose using the optimal colors determined from the isotropic case even when the actual scanning pattern is not isotropic (e.g., covers only part of the sky). We perform simulations showing that maps in multiple colors can be reconstructed accurately from both full-sky and partial-sky scans. Although the original motivation for this research comes from mapping the CMB, this method of polychromatic map-making will have broader applications throughout astrophysics. © 2022, CC BY.

关键词： Color

Algorithmic stability implies training-conditional coverage for distribution-free prediction methods

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Liang, Ruiting Barber, Rina Foygel Committee on Computational and Applied Mathematics University of Chicago United States Department of Statistics University of Chicago United States

In a supervised learning problem, given a predicted value that is the output of some trained model, how can we quantify our uncertainty around this prediction? Distribution-free predictive inference aims to construct prediction intervals around this output, with valid coverage that does not rely on assumptions on the distribution of the data or the nature of the model training algorithm. Existing methods in this area, including conformal prediction and jackknife+, offer theoretical guarantees that hold marginally (i.e., on average over a draw of training and test data). In contrast, training-conditional coverage is a stronger notion of validity that ensures predictive coverage of the test point for most draws of the training data, and is thus a more desirable property in practice. Training-conditional coverage was shown by Vovk [2012] to hold for the split conformal method, but recent work by Bian and Barber [2023] proves that such validity guarantees are not possible for the full conformal and jackknife+ methods without further assumptions. In this paper, we show that an assumption of algorithmic stability ensures that the training-conditional coverage property holds for the full conformal and jackknife+ methods. Copyright © 2023, The Authors. All rights reserved.

关键词： Forecasting

Batched Nonparametric Contextual Bandits

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Jiang, Rong Ma, Cong Committee on Computational and Applied Mathematics University of Chicago United States Department of Statistics University of Chicago United States

We study nonparametric contextual bandits under batch constraints, where the expected reward for each action is modeled as a smooth function of covariates, and the policy updates are made at the end of each batch of observations. We establish a minimax regret lower bound for this setting and propose a novel batch learning algorithm that achieves the optimal regret (up to logarithmic factors). In essence, our procedure dynamically splits the covariate space into smaller bins, carefully aligning their widths with the batch size. Our theoretical results suggest that for nonparametric contextual bandits, a nearly constant number of policy updates can attain optimal regret in the fully online setting. © 2024, CC BY.

关键词： Optimization

ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models

学校读者我要写书评

暂无评论

arXiv 2023年

作者： Parkinson, Suzanna Ongie, Greg Willett, Rebecca Committee on Computational and Applied Mathematics University of Chicago ChicagoIL United States Department of Mathematical and Statistical Sciences Marquette University MilwaukeeWI United States Department of Statistics Department of Computer Science and Committee on Computational and Applied Mathematics University of Chicago ChicagoIL United States

Neural networks often operate in the overparameterized regime, in which there are far more parameters than training samples, allowing the training data to be fit perfectly. That is, training the network effectively learns an interpolating function, and properties of the interpolant affect predictions the network will make on new samples. This manuscript explores how properties of such functions learned by neural networks of depth greater than two layers. Our framework considers a family of networks of varying depths that all have the same capacity but different representation costs. The representation cost of a function induced by a neural network architecture is the minimum sum of squared weights needed for the network to represent the function;it reflects the function space bias associated with the architecture. Our results show that adding additional linear layers to the input side of a shallow ReLU network yields a representation cost favoring functions with low mixed variation – that is, it has limited variation in directions orthogonal to a low-dimensional subspace and can be well approximated by a single- or multi-index model. Such functions may be represented by the composition of a function with low two-layer representation cost and a low-rank linear operator. Our experiments confirm this behavior in standard network training regimes. They additionally show that linear layers can improve generalization and the learned network is well-aligned with the true latent low-dimensional linear subspace when data is generated using a multi-index model. Copyright © 2023, The Authors. All rights reserved.

关键词： Interpolation

Cubic lagrange elements satisfying exact incompressibility

学校读者我要写书评

暂无评论

arXiv 2017年

作者： Guzmán, Johnny Ridgway Scott, L. Division of Applied Mathematics Brown University ProvidenceRI02912 United States Departments of Computer Science and Mathematics Committee on Computational and Applied Mathematics University of Chicago ChicagoIL60637 United States

We prove that an analog of the Scott-Vogelius finite elements are inf-sup stable on certain nondegenerate meshes for piecewise cubic velocity fields. We also characterize the divergence of the velocity space on such meshes. In addition, we show how such a characterization relates to the dimension of C1 piecewise quartics on the same mesh. Copyright © 2017, The Authors. All rights reserved.

关键词： Velocity