检索结果-内蒙古大学图书馆

On the influence of over-parameterization in manifold based surrogates and deep neural operators

JOURNAL OF COMPUTATIONAL PHYSICS 2023年第1期479卷

作者： Kontolati, Katiana Goswami, Somdatta Shields, Michael D. Karniadakis, George Em Johns Hopkins Univ Dept Civil & Syst Engn Baltimore MD USA Brown Univ Div Appl Math Providence RI 02912 USA Brown Univ Sch Engn Providence RI USA

Constructing accurate and generalizable approximators (surrogate models) for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. The main question is what type of surrogate models we should construct and should these models be under-parameterized or over-parameterized. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. While m-PCE enables the construction of a mapping by first identifying low-dimensional embeddings of the input functions, parameters, and quantities of interest (QoIs), a neural operator learns the nonlinear mapping via the use of deep neural networks. We demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of the input functions and the output QoIs. To further enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher approximation accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another recently proposed operator learning m

关键词： surrogate -overparameterization over-parameterization DeepONet Manifold-based polynomial chaos expansion Scientific machine learning Brusselator diffusion-reaction system Neural operators

来源：评论

学校读者我要写书评

暂无评论

Do we really need a new theory to understand over-parameterization?

引用

NEUROCOMPUTING 2023年第1期543卷

作者： Oneto, Luca Ridella, Sandro Anguita, Davide Univ Genoa Via Opera Pia 11a I-16145 Genoa Italy

This century saw an unprecedented increase of public and private investments in Artificial Intelligence (AI) and especially in (Deep) Machine Learning (ML). This led to breakthroughs in their practical ability to solve complex real-world problems impacting research and society at large. Instead, our ability to understand the fundamental mechanism behind these breakthroughs has slowed down because of their increased complexity, while in the past breakthroughs often emerged from foundational research. This questioned researchers about the necessity for a new theoretical framework able to help researchers catch up on this lag. One of the still not well understood mechanisms is the so-called over-parametrization, namely the ability of certain models to increase their generalization performance (re-duce test error) when the number of parameters is above the interpolating threshold (zero training error). In this paper we will show that this phenomenon can be better understood using both known theories (surveying them in the process) and empirical evidences for both shallow and deep learning algorithms. (c) 2023 Elsevier B.V. All rights reserved.

关键词： over-parameterization Generalization Algorithmic stability Shallow learning Deep learning

来源：评论

学校读者我要写书评

暂无评论

Global convergence of sub-gradient method for robust matrix recovery: small initialization, noisy measurements, and over-parameterization

The Journal of Machine Learning Research

引用

The Journal of Machine Learning Research 2023年第1期24卷 4434-4517页

作者： Jianhao Ma Salar Fattahi Department of Industrial and Operations Engineering University of Michigan Ann Arbor MI

In this work, we study the performance of sub-gradient method (SubGM) on a natural nonconvex and nonsmooth formulation of low-rank matrix recovery with ℓ1-loss, where the goal is to recover a low-rank matrix from a limited number of measurements, a subset of which may be grossly corrupted with noise. We study a scenario where the rank of the true solution is unknown and over-estimated instead. The over-estimation of the rank gives rise to an over-parameterized model in which there are more degrees of freedom than needed. Such over-parameterization may lead to overfitting, or adversely affect the performance of the algorithm. We prove that a simple SubGM with small initialization is agnostic to both over-parameterization and noise in the measurements. In particular, we show that small initialization nullifies the effect of over-parameterization on the performance of SubGM, leading to an exponential improvement in its convergence rate. Moreover, we provide the first unifying framework for analyzing the behavior of SubGM under both outlier and Gaussian noise models, showing that SubGM converges to the true solution, even under arbitrarily large and arbitrarily dense noise values, and, perhaps surprisingly, even if the globally optimal solutions do not correspond to the ground truth. At the core of our results is a robust variant of restricted isometry property, called Sign-RIP, which controls the deviation of the sub-differential of the ℓ1-loss from that of an ideal, expected loss. As a byproduct of our results, we consider a subclass of robust low-rank matrix recovery with Gaussian measurements, and show that the number of required samples to guarantee the global convergence of SubGM is independent of the over-parameterized rank.

关键词： low-rank matrix recovery nonconvex optimization nonsmooth optimization sub-gradient method over-parameterization

来源：评论

学校读者我要写书评

暂无评论

Deep phase retrieval: Analyzing over-parameterization in phase retrieval

引用

SIGNAL PROCESSING 2021年 180卷 107866-107866页

作者： Yu, Qi Huang, Jun-Jie Zhu, Jubo Dai, Wei Dragotti, Pier Luigi Natl Univ Def Technol Coll Liberal Arts & Sci Changsha Peoples R China Imperial Coll London Dept Elect & Elect Engn London England

Phase retrieval aims to recover the phase of a complex-valued signal from magnitude measurements. Recently, approaches based on non-convex optimization which solve the phase retrieval problem directly using gradient descent methods have become more and more popular. In this paper, we propose a non convex Deep Phase Retrieval (DeepPR) method which over-parameterizes the decision variable as a series of multiplications between over-parameterized decision variables. Gradient descent is then used to optimize the decision variables. When the step size is assumed to be fixed, theoretical analysis shows that over-parameterization does not complicate the original problem and introduces an implicit acceleration effect for optimization. Numerical tests verify this benefit by showing that our proposed model achieves a faster convergence rate compared to the benchmark algorithm. When spectral method is used to determine the initialization of gradient refinement, our simulation results show that DeepPR method enjoys higher empirical success rate than the benchmark algorithm at low sampling rate condition. (C) 2020 Elsevier B.V. All rights reserved.

关键词： Phase retrieval over-parameterization Non-convex optimization Deep model

来源：评论

学校读者我要写书评

暂无评论

Distributed optimization for degenerate loss functions arising from over-parameterization

引用

ARTIFICIAL INTELLIGENCE 2021年 301卷 103575-103575页

作者： Zhang, Chi Li, Qianxiao Inst High Performance Comp Singapore Singapore Natl Univ Singapore Singapore Singapore

We consider distributed optimization with degenerate loss functions, where the optimal sets of local loss functions have a non-empty intersection. This regime often arises in optimizing large-scale multi-agent AI systems (e.g., deep learning systems), where the number of trainable weights far exceeds the number of training samples, leading to highly degenerate loss surfaces. Under appropriate conditions, we prove that distributed gradient descent in this case converges even when communication is arbitrarily less frequent, which is not the case for non-degenerate loss functions. Moreover, we quantitatively analyze the convergence rate, as well as the communication and computation trade -off, providing insights into designing efficient distributed optimization algorithms. Our theoretical findings are confirmed by both distributed convex optimization and deep learning experiments. (C) 2021 Elsevier B.V. All rights reserved.

关键词： Distributed optimization over-parameterization Deep learning

来源：评论

学校读者我要写书评

暂无评论

Iterative Identification Algorithms for Bilinear-in-parameter Systems by Using the over-parameterization Model and the Decomposition

引用

INTERNATIONAL JOURNAL OF CONTROL AUTOMATION AND SYSTEMS 2018年第6期16卷 2634-2643页

作者： Chen, Mengting Ding, Feng Alsaedi, Ahmed Hayat, Tasawar Jiangnan Univ Minist Educ Sch Internet Things Engn Key Lab Adv Proc Control Light Ind Wuxi 214122 Peoples R China Qingdao Univ Sci & Technol Coll Automat & Elect Engn Qingdao 266061 Peoples R China King Abdulaziz Univ Fac Sci Dept Math NAAM Res Grp Jeddah Saudi Arabia Quaid I Azam Univ Dept Math Islamabad Pakistan

This paper focuses on the identification problem for a class of bilinear-in-parameter systems with an additive noise modeled by an autoregressive moving average process. By using the over-parameterization model, the special form of the bilinear term can be obtained by the model equivalent transformation. Then, we use a decomposition of the model into two synthetic models in order to separate the effect of the two sets of parameters, i.e., the coefficients of the nonlinear basis functions from the parameters of the colored noise. Moreover, two decomposition based iterative algorithms are proposed to identify the unknown parameters. A numerical example is presented to confirm the effectiveness of the proposed methods.

关键词： Bilinear-in-parameter system decomposition iterative identification over-parameterization parameter estimation

来源：评论

学校读者我要写书评

暂无评论

Generalization Performance of Empirical Risk Minimization on over-Parameterized Deep ReLU Nets

引用

IEEE TRANSACTIONS ON INFORMATION THEORY 2025年第3期71卷 1978-1993页

作者： Lin, Shao-Bo Wang, Yao Zhou, Ding-Xuan Xi An Jiao Tong Univ Ctr Intelligent Decis Making & Machine Learning Sch Management Xian 710049 Peoples R China Univ Sydney Sch Math & Stat Sydney NSW 2006 Australia

In this paper, we study the generalization performance of global minima of empirical risk minimization (ERM) on over-parameterized deep ReLU nets. Using a novel deepening scheme for deep ReLU nets, we rigorously prove that there exist perfect global minima achieving optimal generalization error rates for numerous types of data under mild conditions. Since over-parameterization of deep ReLU nets is crucial to guarantee that the global minima of ERM can be realized by the widely used stochastic gradient descent (SGD) algorithm, our results present a potential way to fill the gap between optimization and generalization of deep learning.

关键词： Deep learning Optimization Convergence Approximation algorithms Error analysis Machine learning algorithms Analytical models Sparse matrices Kernel Approximation error empirical risk minimization global minima over-parameterization

来源：评论

学校读者我要写书评

暂无评论

Iterative parameter estimation for a class of fractional-order Hammerstein nonlinear systems disturbed by colored noise

引用

PROCEEDINGS OF THE INSTITUTION OF MECHANICAL ENGINEERS PART I-JOURNAL OF SYSTEMS AND CONTROL ENGINEERING 2025年

作者： Wang, Junwei Ji, Yan Ding, Feng Jiangnan Univ Sch Internet Things Engn Key Lab Adv Proc Control Light Ind Minist Educ 1800 Lihu Lake Rd Wuxi 214122 Peoples R China Qingdao Univ Sci & Technol Coll Automat & Elect Engn Qingdao Peoples R China

Considering the existence of nonlinearity and fractional-order phenomena in practical environments, this paper investigates the parameter estimation methods for a class of fractional-order Hammerstein nonlinear systems disturbed by colored noise. The methods are based on the decomposition strategy, separating the identification of fractional-order from the identification of system parameters. Meanwhile, the system is divided into two subsystems, which are a linear block system and a nonlinear block system by using the hierarchical identification principle. To overcome the problem of redundant parameter estimation, the over-parameterization method and the key item separation method are used, respectively. Then, a two-stage over-parameterization gradient-based iterative algorithm and a key term separation two-stage gradient-based iterative algorithm are derived, which the auxiliary model is used to compute the unmeasurable variables. In addition, we analyze the computational efficiencies of the proposed algorithms. The simulation results indicate that the proposed algorithms are effective. Finally, the proposed method is evaluated through a battery model. The results show well agreement with the real system outputs.

关键词： Parameter estimation key term separation over-parameterization hierarchical identification iterative identification

来源：评论

学校读者我要写书评

暂无评论

Sensor Network Localization via Riemannian Conjugate Gradient and Rank Reduction

引用

IEEE TRANSACTIONS ON SIGNAL PROCESSING 2024年 72卷 1910-1927页

作者： Li, Yicheng Sun, Xinghua Sun Yat Sen Univ Sch Elect & Commun Engn Shenzhen Campus Shenzhen 518107 Peoples R China

This paper addresses the Sensor Network Localization (SNL) problem using received signal strength. The SNL is formulated as an Euclidean Distance Matrix Completion (EDMC) problem under the unit ball sample model. Using the Burer-Monteiro factorization type cost function, the EDMC is solved by Riemannian conjugate gradient with Hager-Zhang line search method on a quotient manifold. A "rank reduction" pre-process is proposed for proper initialization and to achieve global convergence with high probability. Simulations on a synthetic scene show that our approach attains better localization accuracy and is computationally efficient compared to several baseline methods. Characterization of a small local basin of attraction around the global optima of the s-stress function under Bernoulli sampling rule and incoherence matrix completion framework is conducted for the first time. Theoretical result conjectures that the Euclidean distance problem with a structure-less sample mask can be effectively handled using spectral initialization followed by vanilla first-order methods. This preliminary analysis, along with the aforementioned numerical accomplishments, provides insights into revealing the landscape of the s-stress function and may stimulate the design of simpler algorithms to tackle the non-convex formulation of general EDMC problems.

关键词： Euclidean distance matrix completion matrix factorization over-parameterization Riemannian optimization sensor network localization

来源：评论

学校读者我要写书评

暂无评论

Weighted neural tangent kernel: a generalized and improved network-induced kernel

引用

MACHINE LEARNING 2023年第8期112卷 2871-2901页

作者： Tan, Lei Wu, Shutong Zhou, Wenxing Huang, Xiaolin Shanghai Jiao Tong Univ Dept Automat Shanghai Peoples R China

The neural tangent kernel (NTK) has recently attracted intense study, as it describes the evolution of an over-parameterized neural network (NN) trained by gradient descent. However, it is now well-known that gradient descent is not always a good optimizer for NNs, which can partially explain the unsatisfactory practical performance of the NTK regression estimator. In this paper, we introduce the weighted neural tangent kernel (WNTK), a generalized and improved tool, which can capture an over-parameterized NN's training dynamics under adjusted gradient descent direction. Theoretically, in the infinite-width limit, we prove: (1) the stability of the WNTK at initialization and during training, and (2) the equivalence between the WNTK regression estimator and the corresponding NN estimator with different learning rates on different parameters. With the proposed weight update algorithm, weight terms, or equivalently NN descent directions, can be trained through multiple-kernel optimization. Both empirical and analytical WNTKs outperform the corresponding NTKs in numerical experiments, coinciding with the fact that adjusted gradient descent could outperform original gradient descent in NNs' training.

关键词： Neural tangent kernel over-parameterization Adjusted descent direction

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：