检索结果-内蒙古大学图书馆

stochastic Modified Equations and Dynamics of stochastic gradient algorithms I: Mathematical Foundations

JOURNAL OF MACHINE LEARNING RESEARCH 2019年第1期20卷 1474-1520页

作者： Li, Qianxiao Tai, Cheng Weinan, E. Agcy Sci Technol & Res Inst High Performance Comp 1 Fusionopolis Way Singapore 138632 Singapore Beijing Inst Big Data Res Beijing 100080 Peoples R China Peking Univ Beijing 100080 Peoples R China Princeton Univ Princeton NJ 08544 USA

We develop the mathematical foundations of the stochastic modified equations (SME) framework for analyzing the dynamics of stochastic gradient algorithms, where the latter is approximated by a class of stochastic differential equations with small noise parameters. We prove that this approximation can be understood mathematically as an weak approximation, which leads to a number of precise and useful results on the approximations of stochastic gradient descent (SGD), momentum SGD and stochastic Nesterov's accelerated gradient method in the general setting of stochastic objectives. We also demonstrate through explicit calculations that this continuous-time approach can uncover important analytical insights into the stochastic gradient algorithms under consideration that may not be easy to obtain in a purely discrete-time setting.

关键词： stochastic gradient algorithms modified equations stochastic differential equations momentum Nesterov's accelerated gradient

来源：评论

学校读者我要写书评

暂无评论

New Convergence Aspects of stochastic gradient algorithms

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2019年第1期20卷 1-49页

作者： Nguyen, Lam M. Phuong Ha Nguyen Richtarik, Peter Scheinberg, Katya Takac, Martin van Dijk, Marten Thomas J Watson Res Ctr IBM Res Yorktown Hts NY 10598 USA Univ Connecticut Dept Elect & Comp Engn Storrs CT 06268 USA King Abdullah Univ Sci & Technol Comp Elect & Math Sci & Engn Div Thuwal Saudi Arabia Cornell Univ Sch Operat Res & Informat Engn Ithaca NY 14850 USA Lehigh Univ Dept Ind & Syst Engn Bethlehem PA 18015 USA

The classical convergence analysis of SGD is carried out under the assumption that the norm of the stochastic gradient is uniformly bounded. While this might hold for some loss functions, it is violated for cases where the objective function is strongly convex. In Bottou et al. (2018), a new analysis of convergence of SGD is performed under the assumption that stochastic gradients are bounded with respect to the true gradient norm. We show that for stochastic problems arising in machine learning such bound always holds;and we also propose an alternative convergence analysis of SGD with diminishing learning rate regime. We then move on to the asynchronous parallel setting, and prove convergence of Hogwild! algorithm in the same regime in the case of diminished learning rate. It is well-known that SGD converges if a sequence of learning rates {70 satisfies 7 17t 00 and Et o 7/t2 < 00 " We show the convergence of SGD for strongly convex objective function without using bounded gradient assumption when {70 is a diminishing sequence and oo. Inz_,V't 0 other words, we extend the current state-of-the-art class of learning rates satisfying the convergence of SGD.

关键词： stochastic gradient algorithms Asynchronous stochastic Optimization SGD Hogwild bounded gradient

来源：评论

学校读者我要写书评

暂无评论

Projected stochastic gradient Langevin algorithms for Constrained Sampling and Non-Convex Learning 34

Projected Stochastic Gradient Langevin Algorithms for Constr...

引用

Conference on Learning Theory (COLT)

作者： Lamperski, Andrew 200 Union St Se Keller Hall 4-174 Minneapolis MN 55455 USA

Langevin algorithms are gradient descent methods with additive noise. They have been used for decades in Markov Chain Monte Carlo (MCMC) sampling, optimization, and learning. Their convergence properties for unconstrained non-convex optimization and learning problems have been studied widely in the last few years. Other work has examined projected Langevin algorithms for sampling from log-concave distributions restricted to convex compact sets. For learning and optimization, log-concave distributions correspond to convex losses. In this paper, we analyze the case of non-convex losses with compact convex constraint sets and IID external data variables. We term the resulting method the projected stochastic gradient Langevin algorithm (PSGLA). We show the algorithm achieves a deviation of O(T-1/4(log T)(1/2)) from its target distribution in 1-Wasserstein distance. For optimization and learning, we show that the algorithm achieves epsilon-suboptimal solutions, on average, provided that it is run for a time that is polynomial in epsilon(-1) and slightly superexponential in the problem dimension.

关键词： Langevin Methods stochastic gradient algorithms Non-Convex Learning NonAsymptotic Analysis Markov Chain Monte Carlo Sampling

来源：评论

学校读者我要写书评

暂无评论

stochastic modified equations and dynamics of stochastic gradient algorithms I: mathematical foundations

The Journal of Machine Learning Research

引用

The Journal of Machine Learning Research 2019年第1期20卷

作者： Qianxiao Li Cheng Tai E. Weinan Institute of High Performance Computing Agency for Science Technology and Research Connexis North Singapore Beijing Institute of Big Data Research and Peking University Beijing China Princeton University Princeton NJ and Beijing Institute of Big Data Research Beijing China

关键词： Nesterov's accelerated gradient modified equations momentum stochastic differential equations stochastic gradient algorithms

来源：评论

学校读者我要写书评

暂无评论

Estimating the geometric median in Hilbert spaces with stochastic gradient algorithms: L^p and almost sure rates of convergence

引用

JOURNAL OF MULTIVARIATE ANALYSIS 2016年 146卷 209-222页

作者： Godichon-Baggioni, Antoine Univ Bourgogne Inst Math Bourgogne 9 Rue Alain Savary F-21078 Dijon France

The geometric median, also called L-1-median, is often used in robust statistics. Moreover, it is more and more usual to deal with large samples taking values in high dimensional spaces. In this context, a fast recursive estimator has been introduced by Cardot et al. (2013). This work aims at studying more precisely the asymptotic behavior of the estimators of the geometric median based on such non linear stochastic gradient algorithms. The L-p rates of convergence as well as almost sure rates of convergence of these estimators are derived in general separable Hilbert spaces. Moreover, the optimal rates of convergence in quadratic mean of the averaged algorithm are also given. (C) 2015 Elsevier Inc. All rights reserved.

关键词： Functional data analysis Law of large numbers Martingales in Hilbert space Recursive estimation Robust statistics Spatial median stochastic gradient algorithms

来源：评论

学校读者我要写书评

暂无评论

Weighted stochastic gradient Identification algorithms for ARX models

引用

IFAC-PapersOnLine 2015年第28期48卷 1076-1081页

作者： Wu, Ai-Guo Dong, Rui-Qi Fu, Fang-Zhou Harbin Institute of Technology Shenzhen Graduate School Shenzhen518055 China

In this paper, weighted stochastic gradient (WSG) algorithms for ARX models are proposed by modifying the standard stochastic gradient identification algorithms. In the proposed algorithms, the correction term is a weighted term of the correction terms of the standard SG algorithm in the current and last recursive steps. In addition, a latest estimation based WSG (LE-WSG) algorithm is also established. The convergence performance of the proposed LE-WSG algorithm is then analyzed. It is shown by a numerical example that both the WSG and LE-WSG algorithms can possess faster convergence speed and higher convergence precision compared with the standard SG algorithms if the weighting factor is appropriately chosen. © 2015

关键词： stochastic systems Parameter estimation stochastic models ARX model Convergence performance Correction terms Faster convergence Identification algorithms stochastic gradient stochastic gradient algorithms Weighting factors

来源：评论

学校读者我要写书评

暂无评论

SGD with Coordinate Sampling: Theory and Practice

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2022年第1期23卷 1-47页

作者： Leluc, Remi Portier, Francois Inst Polytech Paris Telecom Paris LTCI Dept Stat F-91120 Palaiseau France Ecole Natl Stat & Anal Informat ENSAI Dept Stat CREST F-35170 Bruz France

While classical forms of stochastic gradient descent algorithm treat the different coordinates in the same way, a framework allowing for adaptive (non uniform) coordinate sampling is developed to leverage structure in data. In a non-convex setting and including zeroth-order gradient estimate, almost sure convergence as well as non-asymptotic bounds are established. Within the proposed framework, we develop an algorithm, MUSKETEER, based on a reinforcement strategy: after collecting information on the noisy gradients, it samples the most promising coordinate (all for one);then it moves along the one direction yielding an important decrease of the objective (one for all). Numerical experiments on both synthetic and real data examples confirm the effectiveness of MUSKETEER in large scale problems.

关键词： stochastic optimization stochastic gradient algorithms zeroth-order opti-mization coordinate descent adaptive methods

来源：评论

学校读者我要写书评

暂无评论

An investigation of stochastic trust-region based algorithms for finite-sum minimization

引用

OPTIMIZATION METHODS & SOFTWARE 2024年第5期39卷 937-966页

作者： Bellavia, Stefania Morini, Benedetta Rebegoldi, Simone Univ Firenze Dipartimento Ingn Ind Florence Italy Univ Modena & Reggio Emilia Dipartimento Sci Fis Informat & Matemat Via Giuseppe Campi 213-B Modena Italy

This work elaborates on the TRust-region-ish (TRish) algorithm, a stochastic optimization method for finite-sum minimization problems proposed by Curtis et al. in [F.E. Curtis, K. Scheinberg, and R. Shi, A stochastic trust region algorithm based on careful step normalization, INFORMS. J. Optim. 1(3) (2019), pp. 200-220;F.E. Curtis and R. Shi, A fully stochastic second-order trust region method, Optim. Methods Softw. 37(3) (2022), pp. 844-877]. A theoretical analysis that complements the results in the literature is presented, and the issue of tuning the involved hyper-parameters is investigated. Our study also focuses on a practical version of the method, which computes the stochastic gradient by means of the inner product test and the orthogonality test proposed by Bollapragada et al. in [R. Bollapragada, R. Byrd, and J. Nocedal, Adaptive sampling strategies for stochastic optimization, SIAM. J. Optim. 28(4) (2018), pp. 3312-3343]. It is shown experimentally that this implementation improves the performance of TRish and reduces its sensitivity to the choice of the hyper-parameters.

关键词： Finite-sum minimization stochastic gradient algorithms trust-region adaptive sampling strategies

来源：评论

学校读者我要写书评

暂无评论

A Memory Enhancement Adjustment Method Based on stochastic gradients 41

A Memory Enhancement Adjustment Method Based on Stochastic G...

引用

第41届中国控制会议

作者： Gan Li Key Laboratory of Systems and Control Institute of Systems ScienceAcademy of Mathematics and Systems ScienceChinese Academy of Sciences

ISBN: (数字)9789887581536

ISBN: (纸本)9781665482561

stochastic gradient descent methods and its variants have been widely used to learn the parameters of a neural network by solving an associated non-convex minimization *** propose a new momentum method and adaptive method(MEA/AdaMEA)based on memory enhancement adjustment,which is different from the traditional momentum methods and adaptive *** the theoretical level,we prove the convergence of the MEA method for solving a non-convex minimization problem,and then analyze the generalization error of the MEA method from the perspective of consistent stability,showing that the MEA method can improve the stability of the learned model and enhance the generalization *** the experimental level,we compare the empirical results of the MEA/AdaMEA method and common optimizers for deep learning on the datasets CIFAR10 and CIFAR100 under the network architectures of the convolutional neural networks ResNetl8 and *** demonstrate the effectiveness of our proposed MEA/AdaMEA.

关键词： stochastic gradient algorithms Momentum Adaptive Methods Deep Learning

来源：评论

学校读者我要写书评

暂无评论

Inexact SARAH algorithm for stochastic optimization

引用

OPTIMIZATION METHODS & SOFTWARE 2021年第1期36卷 237-258页

作者： Nguyen, Lam M. Scheinberg, Katya Takac, Martin IBM Res Thomas J Watson Res Ctr Yorktown Hts NY 10598 USA Cornell Univ Sch Operat Res & Informat Engn Ithaca NY USA Lehigh Univ Dept Ind & Syst Engn Bethlehem PA 18015 USA

We develop and analyse a variant of the SARAH algorithm, which does not require computation of the exact gradient. Thus this new method can be applied to general expectation minimization problems rather than only finite sum problems. While the original SARAH algorithm, as well as its predecessor, SVRG, requires an exact gradient computation on each outer iteration, the inexact variant of SARAH (iSARAH), which we develop here, requires only stochastic gradient computed on a mini-batch of sufficient size. The proposed method combines variance reduction via sample size selection and iterative stochastic gradient updates. We analyse the convergence rate of the algorithms for strongly convex and non-strongly convex cases, under smooth assumption with appropriate mini-batch size selected for each case. We show that with an additional, reasonable, assumption iSARAH achieves the best-known complexity among stochastic methods in the case of non-strongly convex stochastic functions.

关键词： stochastic gradient algorithms variance reduction stochastic optimization smooth convex

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：