检索结果-内蒙古大学图书馆

Automatic Adaption of the Step Size in Gradient Descent Training

学校读者我要写书评

暂无评论

Automatic Adaption of the Step Size in Gradient Descent Trai...

2025 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2025

作者： Nogueiras-Rodríguez, Albino Nogueiras-Marco, Ignasi Universitat Politècnica de Catalunya Barcelona Spain Centre d'Estudis Avançats de Blanes - CSIC Blanes Spain

ISBN: (纸本)9798350368741

Selecting an appropriate step size is critical in Gradient Descent algorithms used to train Neural Networks for Deep Learning tasks. A small value of the step size leads to slow convergence, while a large value can lead to instability. In this paper, we present ACME, a gradient descent algorithm that automatically updates the step size to its optimum value. When combined with Adaptive Moment Estimation (ADAM) and/or Adaptive Sharpness Aware Minimization (ASAM), ACME achieves performances similar to them, but without the burden of carefully adjusting the learning rate or designing intricate learning rate scheduling policies. The code of ACME, ready to be used by itself or in combination with ADAM and/or ASAM, is available at https://***/ignasi00/***. © 2025 IEEE.

关键词： Deep Learning Gradient Descent Image Recognition Learning Rate optimization algorithms Step Size

A consensus-based optimization method for nonsmooth nonconvex programs with approximated gradient descent scheme

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Wei, Jiazhen Wu, Fan Bian, Wei School of Mathematics Harbin Institute of Technology Harbin China

In this paper, we are interested in finding the global minimizer of a nonsmooth nonconvex unconstrained optimization problem. By combining the discrete consensus-based optimization (CBO) algorithm and the gradient descent method, we develop a novel CBO algorithm with an extra gradient descent scheme evaluated by the forward-difference technique on the function values, where only the objective function values are used in the proposed algorithm. First, we prove that the proposed algorithm can exhibit global consensus in an exponential rate in two senses and possess a unique global consensus point. Second, we evaluate the error estimate between the objective function value on the global consensus point and its global minimum. In particular, as the parameter β tends to ∞, the error converges to zero and the convergence rate is O (logββ) . Third, under some suitable assumptions on the objective function, we provide the number of iterations required for the mean square error in expectation to reach the desired accuracy. It is worth underlining that the theoretical analysis in this paper does not use the mean-field limit. Finally, we illustrate the improved efficiency and promising performance of our novel CBO method through some experiments on several nonconvex benchmark problems and the application to train deep neural networks. Copyright © 2025, The Authors. All rights reserved.

关键词： optimization algorithms

An Analysis of Safety Guarantees in Multi-Task Bayesian optimization

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Lübsen, Jannis O. Eichler, Annika Institute of Control Systems Hamburg University of Technology Hamburg Germany Deutsches-Elektronen Synchrotron Hamburg Germany

In many practical scenarios of black box optimization, the objective function is subject to constraints that must be satisfied to avoid undesirable outcomes. Such constraints are typically unknown and must be learned during optimization. Safe Bayesian optimization aims to find the global optimum while ensuring that the constraints are satisfied with high probability. However, it is often sample-inefficient due to the small initial feasible set, which requires expansion by evaluating the objective or constraint functions, limiting its applicability to low-dimensional or inexpensive problems. To enhance sample efficiency, additional information from cheap simulations can be leveraged, albeit at the cost of safeness guarantees. This paper introduces a novel safe multi-task Bayesian optimization algorithm that integrates multiple tasks while maintaining high-probability safety. We derive robust uniform error bounds for the multi-task case and demonstrate the effectiveness of the approach on benchmark functions and a control problem. Our results show a significant improvement in sample efficiency, making the proposed method well-suited for expensive-to-evaluate functions. © 2025, CC BY-NC-ND.

关键词： optimization algorithms

A Projected Variable Smoothing for Weakly Convex optimization and Supremum Functions

学校读者我要写书评

暂无评论

arXiv 2025年

作者： López-Rivera, Sergio Pérez-Aros, Pedro Vilches, Emilio Departamento de Ingeniería Matemática Universidad de Chile Santiago Chile Instituto de Ciencias de la Ingeniería Universidad de O’Higgins Rancagua Chile

In this paper, we address two main topics. First, we study the problem of minimizing the sum of a smooth function and the composition of a weakly convex function with a linear operator on a closed vector subspace. For this problem, we propose a projected variable smoothing algorithm and establish a complexity bound of Opϵ´3q to achieve an ϵ-approximate solution. Second, we investigate the Moreau envelope and the proximity operator of functions defined as the supremum of weakly convex functions, and we compute the proximity operator in two important cases. In addition, we apply the proposed algorithm for solving a distributionally robust optimization problem, the LASSO with linear constraints, and the max dispersion problem. We illustrate numerical results for the max dispersion *** Codes 90C2, 49J52, 65K05 © 2025, CC BY.

关键词： optimization algorithms

Fig Tree-Wasp Symbiotic Coevolutionary optimization Algorithm

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Kulkarni, Anand J. Purnapatre, Isha Shastri, Apoorva S. Institute of Artificial Intelligence Dr Vishwanath Karad MIT World Peace University 124 Paud Road Kothrud MH Pune411038 India Department of Computer Engineering Cummins College of Engineering for Women Karvenagar MH Pune411052 India

The nature inspired algorithms are becoming popular due to their simplicity and wider applicability. In the recent past several such algorithms have been developed. They are mainly bio-inspired, swarm based, physics based and socio-inspired;however, the domain based on symbiotic relation between creatures is still to be explored. A novel metaheuristic optimization algorithm referred to as Fig Tree-Wasp Symbiotic Coevolutionary (FWSC) algorithm is proposed. It models the symbiotic coevolutionary relationship between fig trees and wasps. More specifically, the mating of wasps, pollinating the figs, searching for new trees for pollination and wind effect drifting of wasps are modeled in the algorithm. These phenomena help in balancing the two important aspects of exploring the search space efficiently as well as exploit the promising regions. The algorithm is successfully tested on a variety of test problems. The results are compared with existing methods and algorithms. The Wilcoxon Signed Rank Test and Friedman Test are applied for the statistical validation of the algorithm performance. The algorithm is also further applied to solve the real-world engineering problems. The performance of the FWSC underscored that the algorithm can be applied to wider variety of real-world problems. © 2025, CC BY.

关键词： optimization algorithms

Stochastic Gradient Descent for Constrained optimization based on Adaptive Relaxed Barrier Functions

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Dimitrieski, Naum Cao, Jing Ebenbauer, Christian Chair of Intelligent Control Systems RWTH Aachen University Aachen52062 Germany

This paper presents a novel stochastic gradient descent algorithm for constrained optimization. The proposed algorithm randomly samples constraints and components of the finite sum objective function and relies on a relaxed logarithmic barrier function that is appropriately adapted in each optimization iteration. For a strongly convex objective function and affine inequality constraints, step-size rules and barrier adaptation rules are established that guarantee asymptotic convergence with probability one. The theoretical results in the paper are complemented by numerical studies which highlight potential advantages of the proposed algorithm for optimization problems with a large number of constraints. Copyright © 2025, The Authors. All rights reserved.

关键词： optimization algorithms

Alternating direction method of multipliers for polynomial optimization

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Cerone, V. Fosson, S.M. Pirrera, S. Regruto, D. Dipartimento di Automatica e Informatica Politecnico di Torino corso Duca degli Abruzzi 24 Torino10129 Italy

Multivariate polynomial optimization is a prevalent model for a number of engineering problems. From a mathematical viewpoint, polynomial optimization is challenging because it is non-convex. The Lasserre’s theory, based on semidefinite relaxations, provides an effective tool to overcome this issue and to achieve the global optimum. However, this approach can be computationally complex for medium and large scale problems. For this motivation, in this work, we investigate a local minimization approach, based on the alternating direction method of multipliers, which is low-complex, straightforward to implement, and prone to decentralization. The core of the work is the development of the algorithm tailored to polynomial optimization, along with the proof of its convergence. Through a numerical example we show a practical implementation and test the effectiveness of the proposed algorithm with respect to state-of-the-art methodologies. © 2025, CC BY-NC-ND.

关键词： optimization algorithms

BILBO: BILevel Bayesian optimization

学校读者我要写书评

暂无评论

arXiv 2025年

作者： Chew, Ruth Wan Theng Nguyen, Quoc Phong Low, Bryan Kian Hsiang Institute of Data Science National University of Singapore Singapore Applied Artificial Intelligence Institute Deakin University Australia Department of Computer Science National University of Singapore Singapore

Bilevel optimization is characterized by a two-level optimization structure, where the upper-level problem is constrained by optimal lower-level solutions, and such structures are prevalent in real-world problems. The constraint by optimal lower-level solutions poses significant challenges, especially in noisy, constrained, and derivative-free settings, as repeating lower-level optimizations is sample inefficient and predicted lower-level solutions may be suboptimal. We present BILevel Bayesian optimization (BILBO), a novel Bayesian optimization algorithm for general bilevel problems with blackbox functions, which optimizes both upper- and lower-level problems simultaneously, without the repeated lower-level optimization required by existing methods. BILBO samples from confidence-bounds based trusted sets, which bounds the suboptimality on the lower level. Moreover, BILBO selects only one function query per iteration, where the function query selection strategy incorporates the uncertainty of estimated lower-level solutions and includes a conditional reassignment of the query to encourage exploration of the lower-level objective. The performance of BILBO is theoretically guaranteed with a sublinear regret bound for commonly used kernels and is empirically evaluated on several synthetic and real-world problems. © 2025, CC BY.

关键词： optimization algorithms

A Dynamic Mission Planning Method of Multi-Satellite Cooperative Observation for Highly Time-Sensitive Targets 15

学校读者我要写书评

暂无评论

A Dynamic Mission Planning Method of Multi-Satellite Coopera...

15th Asia Conference on Mechanical and Aerospace Engineering, ACMAE 2024

作者： Jiaxin, Hu Demin, Zhao Yanwei, Zhu College of Aerospace Science National University of Defense Technology Changsha410000 China DFH Satellite Co. Ltd. Beijing10094 China

ISBN: (纸本)9781643685892

To address the challenges posed by highly time-sensitive targets with uncertainty and unpredictability in multi-satellite cooperative observation, conventional intelligent optimization algorithms often suffer from time-consuming and unstable issues. This paper presents a novel dynamic mission planning method that cleverly incorporates waiting time for tracking and tracked time as heuristics to guide task assignment. Furthermore, plannable and robustness indicators are introduced to facilitate global trade-offs in resource conflicts among multiple targets. Initially, the constraints specifically suitable for time-sensitive target observation are analyzed, followed by the modeling of the planning problem. Subsequently, a dynamic mission planning approach is proposed that utilizes dynamic priorities, offering a linear time solution for task assignment. Through simulation verification, the efficacy of the proposed method is demonstrated, showcasing its ability to enhance multi-satellite cooperative observation effectively. Additionally, it maximizes resource utilization in the face of diverse conflicts. © 2025 The Authors.

关键词： optimization algorithms