检索结果-内蒙古大学图书馆

Does the Data Induce Capacity Control in Deep Learning?

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Yang, Rubing Mao, Jialin Chaudhari, Pratik Applied Mathematics and Computational Sciences University of Pennsylvania United States Electrical and Systems Engineering University of Pennsylvania United States Computer and Information Science University of Pennsylvania United States

We show that the input correlation matrix of typical classification datasets has an eigenspectrum where, after a sharp initial drop, a large number of small eigenvalues are distributed uniformly over an exponentially large range. This structure is mirrored in a network trained on this data: we show that the Hessian and the Fisher Information Matrix (FIM) have eigenvalues that are spread uniformly over exponentially large ranges. We call such eigenspectra "sloppy" because sets of weights corresponding to small eigenvalues can be changed by large magnitudes without affecting the loss. Networks trained on atypical datasets with non-sloppy inputs do not share these traits and deep networks trained on such datasets generalize poorly. Inspired by this, we study the hypothesis that sloppiness of inputs aids generalization in deep networks. We show that if the Hessian is sloppy, we can compute non-vacuous PAC-Bayes generalization bounds analytically. By exploiting our empirical observation that training predominantly takes place in the non-sloppy subspace of the FIM, we develop data-distribution dependent PAC-Bayes priors that lead to accurate generalization bounds using numerical optimization. © 2021, CC BY-NC-SA.

关键词： Fisher information matrix

A parallel-in-time two-sided preconditioning for all-at-once system from a non-local evolutionary equation with weakly singular kernel

学校读者我要写书评

暂无评论

arXiv 2021年

作者： Lin, Xue-Lei Ng, Michael K. Zhi, Yajing Shenzhen JL Computational Science and Applied Research Institute Shenzhen China Beijing Computational Science Research Center Beijing100193 China Department of Mathematics The University of Hong Kong Pokfulam Hong Kong Department of Computer Science The University of Hong Kong Pokfulam Hong Kong

In this paper, we study a parallel-in-time (PinT) algorithm for all-at-once system from a non-local evolutionary equation with weakly singular kernel where the temporal term involves a non-local convolution with a weakly singular kernel and the spatial term is the usual Laplacian operator with variable coefficients. Such a problem has been intensively studied in recent years thanks to the numerous real world applications. However, due to the non-local property of the time evolution, solving the equation in PinT manner is difficult. We propose to use a two-sided preconditioning technique for the all-at-once discretization of the equation. Our preconditioner is constructed by replacing the variable diffusion coefficients with a constant coefficient to obtain a constant-coefficient all-at-once matrix. We split a square root of the constant Laplacian operator out of the constant-coefficient all-at-once matrix as a right preconditioner and take the remaining part as a left preconditioner, which constitutes our two-sided preconditioning. Exploiting the diagonalizability of the constant-Laplacian matrix and the triangular Toeplitz structure of the temporal discretization matrix, we obtain efficient representations of inverses of the right and the left preconditioners, because of which the iterative solution can be fast updated in a PinT manner. Theoretically, the condition number of the two-sided preconditioned matrix is proven to be uniformly bounded by a constant independent of the matrix size. To the best of our knowledge, for the non-local evolutionary equation with variable coefficients, this is the first attempt to develop a PinT preconditioning technique that has fast and exact implementation and that the corresponding preconditioned system has a uniformly bounded condition number. Numerical results are reported to confirm the efficiency of the proposed two-sided preconditioning *** Codes 65F10, 65F08, 15A12, 15A60 Copyright © 2021, The Authors. All rights

关键词： Matrix algebra

Second-order nonlinear Hall effect in Weyl semimetals

学校读者我要写书评

暂无评论

Physical Review B 2020年第24期102卷 245116-245116页

作者： Yang Gao Furu Zhang Wei Zhang Institute of Applied Physics and Computational Mathematics Beijing 100088 China College of Computer Science and Engineering Anhui University of Science and Technology Huainan 232001 China

We investigate the second-order nonlinear Hall effect of Weyl semimetals using a quantum theory, paying attention to the interband process and quantum interference effect. The Drude-type nonlinear Hall conductivity due to the intraband process is found for each Weyl node in the low-frequency regime (the photon energy of the incident field is much smaller than the chemical potential). While in the high-frequency regime with the interband process dominated, the nonlinear photoconductivity shows a two-photon resonant peak or double peaks due to inversion symmetry breaking. The peak splitting is determined by the chemical potential and tilt of the Weyl node. There is a nonmonotonic dependence of the nonlinear Hall current on the tilt of the Weyl node, and an optimal tilt for the maximum Hall current has been found. Moreover, with the help of quantum interference, the nonlinear Hall response can be effectively modulated by tuning the amplitude, phase, and polarization of the incident fields. In particular, in the presence of driving fields with frequencies ω0 and 3ω0, the interplay between the optical processes of second-harmonic generation and different frequency generation leads to the chirality-selective generation of nonlinear Hall current under suitable conditions.

关键词： Hall effect Optical conductivity Photoconductivity Weyl semimetal Linear response theory

Unbiased Estimation Using a Class of Diffusion Processes

学校读者我要写书评

暂无评论

SSRN

SSRN 2022年

作者： Ruzayqat, Hamza Beskos, Alexandros Crisan, Dan Jasra, Ajay Kantas, Nikolas Applied Mathematics and Computational Science Program Computer Electrical and Mathematical Sciences and Engineering Division King Abdullah University of Science and Technology Thuwal23955-6900 Saudi Arabia Department of Statistical Science University College London LondonWC1E 6BT United Kingdom Department of Mathematics Imperial College London LondonSW7 2AZ United Kingdom

We study the problem of unbiased estimation of expectations with respect to (w.r.t.) π a given, general probability measure on (Rd,B(Rd)) that is absolutely continuous with respect to a standard Gaussian measure. We focus on simulation associated to a particular class of diffusion processes, sometimes termed the Schrödinger-Föllmer Sampler, which is a simulation technique that approximates the law of a particular diffusion bridge process {Xt}t∈[0,1] on Rd, d ∈ N0. This latter process is constructed such that, starting at X0 = 0, one has X1 ∼ π. Typically, the drift of the diffusion is intractable and, even if it were not, exact sampling of the associated diffusion is not possible. As a result, [10, 15] consider a stochastic Euler-Maruyama scheme that allows the development of biased estimators for expectations w.r.t. π. We show that for this methodology to achieve a mean square error of O(Ε2), for arbitrary Ε > 0, the associated cost is O(Ε−5). We then introduce an alternative approach that provides unbiased estimates of expectations w.r.t. π, that is, it does not suffer from the time discretization bias or the bias related with the approximation of the drift function. We prove that to achieve a mean square error of O(Ε2), the associated cost is, with high probability, O(Ε−2|log(Ε)|2+δ), for any δ > 0. We implement our method on several examples including Bayesian inverse problems. © 2022, The Authors. All rights reserved.

关键词： Diffusion

Reliable extrapolation of deep neural operators informed by physics or sparse observations

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Zhu, Min Zhang, Handi Jiao, Anran Karniadakis, George Em Lu, Lu Department of Chemical and Biomolecular Engineering University of Pennsylvania PhiladelphiaPA19104 United States Graduate Group in Applied Mathematics and Computational Science University of Pennsylvania PhiladelphiaPA19104 United States Department of Computer and Information Science University of Pennsylvania PhiladelphiaPA19104 United States Division of Applied Mathematics Brown University ProvidenceRI02912 United States School of Engineering Brown University ProvidenceRI02912 United States

Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. As promising surrogate solvers of partial differential equations (PDEs) for real-time prediction, deep neural operators such as deep operator networks (DeepONets) provide a new simulation paradigm in science and engineering. Pure data-driven neural operators and deep learning models, in general, are usually limited to interpolation scenarios, where new predictions utilize inputs within the support of the training set. However, in the inference stage of real-world applications, the input may lie outside the support, i.e., extrapolation is required, which may result to large errors and unavoidable failure of deep learning models. Here, we address this challenge of extrapolation for deep neural operators. First, we systematically investigate the extrapolation behavior of DeepONets by quantifying the extrapolation complexity via the 2-Wasserstein distance between two function spaces and propose a new behavior of bias-variance trade-off for extrapolation with respect to model capacity. Subsequently, we develop a complete workflow, including extrapolation determination, and we propose five reliable learning methods that guarantee a safe prediction under extrapolation by requiring additional information—the governing PDEs of the system or sparse new observations. The proposed methods are based on either fine-tuning a pre-trained DeepONet or multifidelity learning. We demonstrate the effectiveness of the proposed framework for various types of parametric PDEs. Our systematic comparisons provide practical guidelines for selecting a proper extrapolation method depending on the available information, desired accuracy, and required inference speed. © 2022, CC BY-NC-ND.

关键词： Extrapolation

KALEIDOSCOPE: AN EFFICIENT, LEARNABLE REPRESENTATION FOR ALL STRUCTURED LINEAR MAPS 8

学校读者我要写书评

暂无评论

KALEIDOSCOPE: AN EFFICIENT, LEARNABLE REPRESENTATION FOR ALL...

8th International Conference on Learning Representations, ICLR 2020

作者： Dao, Tri Sohoni, Nimit Sharad Gu, Albert Eichhorn, Matthew Blonder, Amit Leszczynski, Megan Rudra, Atri Ré, Christopher Department of Computer Science Stanford University United States Institute for Computational and Mathematical Engineering Stanford University United States Center for Applied Mathematics Cornell University United States Department of Computer Science and Engineering University at Buffalo The State University of New York United States

Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps. However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy. We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity. We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality. For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%. K-matrices can also simplify hand-engineered pipelines-we replace filter bank feature computation in speech data preprocessing with a learnable kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task. In addition, K-matrices can capture latent structure in models: for a challenging permuted image classification task, a K-matrix based representation of permutations is able to learn the right latent structure and improves accuracy of a downstream convolutional model by over 9%. We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task. © 2020 8th International Conference on Learning Representations, ICLR 2020. All rights reserved.

关键词： Matrix algebra

Preconditioning for the integration of a spatiotemporal pharmacodynamic system

学校读者我要写书评

暂无评论

Preconditioning for the integration of a spatiotemporal phar...

92nd Annual Meeting of the International Association of applied mathematics and Mechanics, GAMM 2022

作者： Tebbens, Jurjen Duintjer Lanzendörfer, Martin Matonoha, Ctirad Papáček, Štěpán Department of Computational Mathematics Institute of Computer Science of the Czech Academy of Sciences Pod Vodárenskou věží 271/2 Praha-Libeň 182 07 Czech Republic Department of Biophysics and Physical Chemistry Faculty of Pharmacy in Hradec Kralove Charles University Heyrovského 1203 Hradec Králové500 05 Czech Republic Institute of Hydrogeology Engi neering Geology and Applied Geophysics Faculty of Science Charles University Albertov 6 Praha 2 128 43 Czech Republic Department of Control Theory Institute of Information Theory and Automation of the Czech Academy of Sciences Pod Vodárenskou věží 4 Praha-Libeň 182 00 Czech Republic

We address efficient modeling of the intracellular action of a drug binding to the nuclear pregnane X receptor. The binded complex enters the cell nucleus and acts on DNA, resulting in enhanced production of an enzyme which metabolizes, among others, co-administered drugs. Pharmacodynamic modeling is traditionally based on a compartmental approach, which simplifies the complex processes in the human body through the definition of a small number of compartments representing organs, tissues, cells, cytoplasm or abstract units. Inside a compartment, the concentrations of all substances are assumed to be distributed homogeneously, i.e. there is no spatial dependence. They lead to a system of ODEs for the time-dependence of the concentrations of the active substances. We will present an extension of a model from the literature for the action of the tuberculose-drug Rifampicin. The extension consists of a first attempt to add spatial resolution for substances that are active in the cytoplasm. Spatial resolution can be benecifial to analyze important issues like, among others, local exceeding of toxic drug levels, delay of transport and drug-drug interactions. We address efficient solution of the linear systems arising when numerically integrating the resulting PDE’s and consider some preconditioning techniques based on properties of the underlying biochemical network. © 2023 Wiley-VCH GmbH.

关键词： Linear systems

The Role of Heterogeneity in Autonomous Perimeter Defense Problems

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Adler, Aviv Mickelin, Oscar Ramachandran, K. Sukhatme, Gaurav S. Karaman, Sertac Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology CambridgeMA United States Program in Applied and Computational Mathematics Princeton University Princeton NJ08544 United States Department of Computer Science University of Southern California Los AngelesCA90089 United States Department of Aeronautics and Astronautics Massachusetts Institute of Technology CambridgeMA02139 United States

When is heterogeneity in the composition of an autonomous robotic team beneficial and when is it detrimental? We investigate and answer this question in the context of a minimally viable model that examines the role of heterogeneous speeds in perimeter defense problems, where defenders share a total allocated speed budget. We consider two distinct problem settings and develop strategies based on dynamic programming and on local interaction rules. We present a theoretical analysis of both approaches and our results are extensively validated using simulations. Interestingly, our results demonstrate that the viability of heterogeneous teams depends on the amount of information available to the defenders. Moreover, our results suggest a universality property: across a wide range of problem parameters the optimal ratio of the speeds of the defenders remains nearly constant. Copyright © 2022, The Authors. All rights reserved.

关键词： Dynamic programming

Unbiased Estimation using a Class of Diffusion Processes

学校读者我要写书评

暂无评论

arXiv 2022年

作者： Ruzayqat, Hamza Beskos, Alexandros Crisan, Dan Jasra, Ajay Kantas, Nikolas Applied Mathematics and Computational Science Program Computer Electrical and Mathematical Sciences and Engineering Division King Abdullah University of Science and Technology Thuwal23955-6900 KSA Saudi Arabia Department of Statistical Science University College London LondonWC1E 6BT United Kingdom Department of Mathematics Imperial College London LondonSW7 2AZ United Kingdom

We study the problem of unbiased estimation of expectations with respect to (w.r.t.) π a given, general probability measure on (d, B(d)) that is absolutely continuous with respect to a standard Gaussian measure. We focus on simulation associated to a particular class of diffusion processes, sometimes termed the Schrödinger-Föllmer Sampler, which is a simulation technique that approximates the law of a particular diffusion bridge process {Xt}tϵ[0,1]on d, d ϵ 0. This latter process is constructed such that, starting at X0= 0, one has X1∼ π. Typically, the drift of the diffusion is intractable and, even if it were not, exact sampling of the associated diffusion is not possible. As a result, [10, 16] consider a stochastic Euler-Maruyama scheme that allows the development of biased estimators for expectations w.r.t. π. We show that for this methodology to achieve a mean square error of O(ϵ2), for arbitrary ϵ > 0, the associated cost is O(ϵ-5). We then introduce an alternative approach that provides unbiased estimates of expectations w.r.t. π, that is, it does not suffer from the time discretization bias or the bias related with the approximation of the drift function. We prove that to achieve a mean square error of O(ϵ2), the associated cost (which is random) is, with high probability, O(ϵ-2| log(ϵ)|2+δ), for any δ > 0. We implement our method on several examples including Bayesian inverse *** Codes 60J60, 62D05, 65C40 © 2022, CC BY.

关键词： Diffusion