检索结果-内蒙古大学图书馆

A Generalization Result for Convergence in Learning-to-Optimize

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Sucker, Michael Ochs, Peter Department of Mathematics University of Tübingen Tübingen Germany Department of Mathematics and Computer Science Saarland University Saarbrücken Germany

Convergence in learning-to-optimize is hardly studied, because conventional convergence guarantees in optimization are based on geometric arguments, which cannot be applied easily to learned algorithms. Thus, we develop a probabilistic framework that resembles deterministic optimization and allows for transferring geometric arguments into learning-to-optimize. Our main theorem is a generalization result for parametric classes of potentially non-smooth, non-convex loss functions and establishes the convergence of learned optimization algorithms to stationary points with high probability. This can be seen as a statistical counterpart to the use of geometric safeguards to ensure convergence. To the best of our knowledge, we are the first to prove convergence of optimization algorithms in such a probabilistic framework. © 2024, CC BY.

关键词： optimization algorithms

COMPLEXITY AND NUMERICAL EXPERIMENTS OF A NEW ADAPTIVE GENERIC PROXIMAL BUNDLE METHOD

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Guigues, Vincent Monteiro, Renato D.C. Tran, Benoit School of Applied Mathematics FGV Praia de Botafogo Rio de Janeiro Brazil Georgia Institute of Technology AtlantaGA30332-0205 United States

This paper develops an adaptive generic proximal bundle method, shows its complexity, and presents numerical experiments comparing this method with two bundle methods on a set of optimization problems. Copyright ©... 详细信息

关键词： optimization algorithms

The Implicit Bias of Gradient Descent on Separable Multiclass Data

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ravi, Hrithik Scott, Clayton Soudry, Daniel Wang, Yutong University of Michigan United States Technion - Israel Institute of Technology Illinois Institute of Technology United States

Implicit bias describes the phenomenon where optimization-based training algorithms, without explicit regularization, show a preference for simple estimators even when more complex estimators have equal objective values. Multiple works have developed the theory of implicit bias for binary classification under the assumption that the loss satisfies an exponential tail property. However, there is a noticeable gap in analysis for multiclass classification, with only a handful of results which themselves are restricted to the cross-entropy loss. In this work, we employ the framework of Permutation Equivariant and Relative Margin-based (PERM) losses [Wang and Scott, 2024] to introduce a multiclass extension of the exponential tail property. This class of losses includes not only cross-entropy but also other losses. Using this framework, we extend the implicit bias result of Soudry et al. [2018] to multiclass classification. Furthermore, our proof techniques closely mirror those of the binary case, thus illustrating the power of the PERM framework for bridging the binary-multiclass gap. © 2024, CC BY-NC-SA.

关键词： optimization algorithms

Improved SQP and SLSQP algorithms for Feasible Path-based Process Optimisation

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Ma, Yingjie Gao, Xi Liu, Chao Li, Jie Centre for Process Integration Department of Chemical Engineering School of Engineering The University of Manchester ManchesterM13 9PL United Kingdom Department of Chemical Engineering Massachusetts Institute of Technology CambridgeMA02139 United States School of Electronic and Information Engineering Tongji University Shanghai201804 China School of Mechanical and Electrical Engineering Jinggangshan University Jiangxi Ji'an343009 China

Feasible path algorithms have been widely used for process optimisation due to its good convergence. The sequential quadratic programming (SQP) algorithm is usually used to drive the feasible path algorithms towards optimality. However, existing SQP algorithms may suffer from inconsistent quadratic programming (QP) subproblems and numerical noise, especially for ill-conditioned process optimisation problems, leading to a suboptimal or infeasible solution. In this work, we propose an improved SQP algorithm (I-SQP) and an improved sequential least squares programming algorithm (I-SLSQP) that solves a least squares (LSQ) subproblem at each major iteration. A hybrid method through the combination of two existing relaxations is proposed to solve the inconsistent subproblems for better convergence and higher efficiency. We find that a certain part of the dual LSQ algorithm suffers from serious cancellation errors, resulting in an inaccurate search direction or no viable search direction generated. Therefore, the QP solver is used to solve LSQ subproblems in such a situation. The computational results indicates that I-SLSQP is more robust than fmincon in MATLAB, IPOPT, Py-SLSQP and I-SQP. It is also shown that I-SLSQP and Py-SLSQP is superior to I-SQP for ill-conditioned process optimisation problems, whilst I-SQP is more computationally efficient than I-SLSQP and Py-SLSQP for well-conditioned problems. © 2024, CC BY-NC-ND.

关键词： optimization algorithms

Computing Optima in Standard Quadratic Functions in Zero-Error Probabilistic Polynomial Time

学校读者我要写书评

暂无评论

SSRN

SSRN 2024年

作者： Gao, Lunshan Physics and Computer Science Wilfrid Laurier University 75 University Ave. W. WaterlooONN2L3C5 Canada

Fuzzy numbers are used to represent uncertainties arising from imprecision and vagueness. In this paper, we propose a novel randomized algorithm for computing local optima in standard quadratic optimization problems by using fuzzification, random number generators, and the first and second order conditions. This paper proves that: (1) the computational complexity of the new algorithm is in zero-error probabilistic polynomial time;(2) the transformation used in the new algorithm is polynomial time reduction. Examplesand numerical experiments show that the new algorithm outperforms both Gurobi optimizer and IBM CPLEX solvers and the new algorithm is faster than both Gurobi optimizer and IBM CPLEX solvers. © 2024, The Authors. All rights reserved.

关键词： optimization algorithms

Functionally Constrained Algorithm Solves Convex Simple Bilevel Problems

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Zhang, Huaqing Chen, Lesi Xu, Jing Zhang, Jingzhao IIIS Tsinghua University China Shanghai Qizhi Institute China Shanghai AI Lab China

This paper studies simple bilevel problems, where a convex upper-level function is minimized over the optimal solutions of a convex lower-level problem. We first show the fundamental difficulty of simple bilevel problems, that the approximate optimal value of such problems is not obtainable by first-order zero-respecting algorithms. Then we follow recent works to pursue the weak approximate solutions. For this goal, we propose a novel method by reformulating them into functionally constrained problems. Our method achieves near-optimal rates for both smooth and nonsmooth problems. To the best of our knowledge, this is the first near-optimal algorithm that works under standard assumptions of smoothness or Lipschitz continuity for the objective functions. Copyright © 2024, The Authors. All rights reserved.

关键词： optimization algorithms

Towards optimal hierarchical training of neural networks

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Feischl, Michael Rieder, Alexander Zehetgruber, Fabian Institute of Analysis and Scientific Computing TU Wien Austria

We propose a hierarchical training algorithm for standard feed-forward neural networks that adaptively extends the network architecture as soon as the optimization reaches a stationary point. By solving small (low-dimensional) optimization problems, the extended network provably escapes any local minimum or stationary point. Under some assumptions on the approximability of the data with stable neural networks, we show that the algorithm achieves an optimal convergence rate s in the sense that loss #parameters−s. As a byproduct, we obtain computable indicators which judge the optimality of the training state of a given network and derive a new notion of generalization error. Copyright © 2024, The Authors. All rights reserved.

关键词： optimization algorithms

A Full Adagrad algorithm with O(Nd) operations

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Godichon-Baggioni, Antoine Lu, Wei Portier, Bruno Sorbonne Université 4 Place Jussieu Paris75005 France Laboratoire de Mathématiques INSA Rouen Normandie BP 08 - Avenue de l’Université Saint-Etienne du Rouvray76800 France

A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By developing a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applications. This innovative strategy significantly reduces the complexity and resource demands typically associated with full-matrix methods, enabling more effective optimization processes. Moreover, the convergence rates of the proposed estimators and their asymptotic efficiency are given. Their effectiveness is demonstrated through numerical studies. Copyright © 2024, The Authors. All rights reserved.

关键词： optimization algorithms

Anderson Acceleration in Nonsmooth Problems: Local Convergence via Active Manifold Identification

学校读者我要写书评

暂无评论

arXiv 2024年

作者： Li, Kexin Bai, Luwei Wang, Xiao Wang, Hao School of Information Science and Technology ShanghaiTech University Shanghai China Department of AI Computing Peng Cheng Laboratory Shenzhen China

Anderson acceleration is an effective technique for enhancing the efficiency of fixed-point iterations;however, analyzing its convergence in nonsmooth settings presents significant challenges. In this paper, we investigate a class of nonsmooth optimization algorithms characterized by the active manifold identification property. This class includes a diverse array of methods such as the proximal point method, proximal gradient method, proximal linear method, proximal coordinate descent method, Douglas-Rachford splitting (or the alternating direction method of multipliers), and the iteratively reweighted 1 method, among others. Under the assumption that the optimization problem possesses an active manifold at a stationary point, we establish a local R-linear convergence rate for the Anderson-accelerated algorithm. Our extensive numerical experiments further highlight the robust performance of the proposed Anderson-accelerated methods. Copyright © 2024, The Authors. All rights reserved.

关键词： optimization algorithms