检索结果-内蒙古大学图书馆

actor-critic Learning algorithms for Mean-Field Control with Moment Neural Networks

METHODOLOGY AND COMPUTING IN APPLIED PROBABILITY 2025年第1期27卷 1-20页

作者： Pham, Huyen Warin, Xavier Univ Paris Cite Paris France FiME Paris France Ecole Polytech CMAP F-91128 Palaiseau France EDF R&D FiME Paris France

We develop a new policy gradient and actor-critic algorithm for solving mean-field control problems within a continuous time reinforcement learning setting. Our approach leverages a gradient-based representation of the value function, employing parametrized randomized policies. The learning for both the actor (policy) and critic (value function) is facilitated by a class of moment neural network functions on the Wasserstein space of probability measures, and the key feature is to sample directly trajectories of distributions. A central challenge addressed in this study pertains to the computational treatment of an operator specific to the mean-field framework. To illustrate the effectiveness of our methods, we provide a comprehensive set of numerical results. These encompass diverse examples, including multi-dimensional settings and nonlinear quadratic mean-field control problems with controlled volatility.

关键词： Mean-field control Reinforcement learning Policy gradient Moment neural network actor-critic algorithms

来源：评论

学校读者我要写书评

暂无评论

Bayesian Policy Gradient and actor-critic algorithms

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2016年第1期17卷 2319-2371页

作者： Ghavamzadeh, Mohammad Engel, Yaakov Valko, Michal Adobe Res Cambridge MA USA INRIA Paris France Rafael Adv Def Syst Tel Aviv Israel INRIA Lille SequeL Team Lille France

Policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate. Many conventional policy gradient methods use Monte-Carlo techniques to estimate this gradient. The policy is improved by adjusting the parameters in the direction of the gradient estimate. Since Monte-Carlo methods tend to have high variance, a large number of samples is required to attain accurate estimates, resulting in slow convergence. In this paper, we fi rst propose a Bayesian framework for policy gradient, based on modeling the policy gradient as a Gaussian process. This reduces the number of samples needed to obtain accurate gradient estimates. Moreover, estimates of the natural gradient as well as a measure of the uncertainty in the gradient estimates, namely, the gradient covariance, are provided at little extra cost. Since the proposed Bayesian framework considers system trajectories as its basic observable unit, it does not require the dynamics within trajectories to be of any particular form, and thus, can be easily extended to partially observable problems. On the downside, it cannot take advantage of the Markov property when the system is Markovian. To address this issue, we proceed to supplement our Bayesian policy gradient framework with a new actor-critic learning model in which a Bayesian class of non-parametric critics, based on Gaussian process temporal di ff erence learning, is used. Such critics model the action-value function as a Gaussian process, allowing Bayes' rule to be used in computing the posterior distribution over action-value functions, conditioned on the observed data. Appropriate choices of the policy parameterization and of the prior covariance (kernel) between action-values allow us to obtain closed-form expressions for the posterior distribution of the gradient of the expected return with respect to the policy parameters. We perform detailed experimental comparisons of the proposed Baye

关键词： reinforcement learning policy gradient methods actor-critic algorithms Bayesian inference Gaussian processes

来源：评论

学校读者我要写书评

暂无评论

On actor-critic algorithms

引用

SIAM JOURNAL ON CONTROL AND OPTIMIZATION 2003年第4期42卷 1143-1166页

作者： Konda, VR Tsitsiklis, JN MIT Informat & Decis Syst Lab Cambridge MA 02139 USA

In this article, we propose and analyze a class of actor-critic algorithms. These are two-time-scale algorithms in which the critic uses temporal difference learning with a linearly parameterized approximation architecture, and the actor is updated in an approximate gradient direction, based on information provided by the critic. We show that the features for the critic should ideally span a subspace prescribed by the choice of parameterization of the actor. We study actor-critic algorithms for Markov decision processes with Polish state and action spaces. We state and prove two results regarding their convergence.

关键词： reinforcement learning Markov decision processes actor-critic algorithms stochastic approximation

来源：评论

学校读者我要写书评

暂无评论

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

引用

MACHINE LEARNING 2016年第3期105卷 367-417页

作者： Prashanth, L. A. Ghavamzadeh, Mohammad Univ Maryland Syst Res Inst College Pk MD 20742 USA Adobe Res San Jose CA USA INRIA Lille France

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales-a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

关键词： Markov decision process (MDP) Reinforcement learning (RL) Risk sensitive RL actor-critic algorithms Multi-time-scale stochastic approximation Simultaneous perturbation stochastic approximation (SPSA) Smoothed functional (SF)

来源：评论

学校读者我要写书评

暂无评论

actor-critic algorithms with Online Feature Adaptation

引用

ACM TRANSACTIONS ON MODELING AND COMPUTER SIMULATION 2016年第4期26卷 24-24页

作者： Prabuchandran, K. J. Bhatnagar, Shalabh Borkar, Vivek S. Indian Inst Sci Dept Comp Sci & Automat Bangalore 560012 Karnataka India Indian Inst Technol Dept Elect Engn Mumbai 400076 Maharashtra India

We develop two new online actor-critic control algorithms with adaptive feature tuning for Markov Decision Processes (MDPs). One of our algorithms is proposed for the long-run average cost objective, while the other works for discounted cost MDPs. Our actor-critic architecture incorporates parameterization both in the policy and the value function. A gradient search in the policy parameters is performed to improve the performance of the actor. The computation of the aforementioned gradient, however, requires an estimate of the value function of the policy corresponding to the current actor parameter. The value function, on the other hand, is approximated using linear function approximation and obtained from the critic. The error in approximation of the value function, however, results in suboptimal policies. In our article, we also update the features by performing a gradient descent on the Grassmannian of features to minimize a mean square Bellman error objective in order to find the best features. The aim is to obtain a good approximation of the value function and thereby ensure convergence of the actor to locally optimal policies. In order to estimate the gradient of the objective in the case of the average cost criterion, we utilize the policy gradient theorem, while in the case of the discounted cost objective, we utilize the simultaneous perturbation stochastic approximation (SPSA) scheme. We prove that our actor-critic algorithms converge to locally optimal policies. Experiments on two different settings show performance improvements resulting from our feature adaptation scheme.

关键词： Markov decision processes actor-critic algorithms function approximation feature adaptation online learning residual gradient scheme temporal difference learning stochastic approximation Grassmann manifold SPSA policy gradients

来源：评论

学校读者我要写书评

暂无评论

Parametrized actor-critic algorithms for finite-horizon MDPs

Parametrized actor-critic algorithms for finite-horizon MDPs

引用

26th American Control Conference

作者： Abdulla, Mohammed Shahid Bhatnagar, Shalabh Indian Inst Sci Dept Comp Sci & Automat Bangalore 560012 Karnataka India

ISBN: (纸本)9781424409884

Due to their non-stationarity, finite-horizon Markov decision processes (FH-MDPs) have one probability transition matrix per stage. Thus the curse of dimensionality affects FH-MDPs more severely than infinite-horizon MDPs. We propose two parametrized 'actor-critic' algorithms to compute optimal policies for FH-MDPs. Both algorithms use the two-timescale stochastic approximation technique, thus simultaneously performing gradient search in the parametrized policy space (the 'actor') on a slower timescale and learning the policy gradient (the 'critic') via a faster recursion. This is in contrast to methods where critic recursions learn the cost-to-go proper. We show w.p 1 convergence to a set with the necessary condition for constrained optima. The proposed parameterization is for FHMDPs with compact action sets, although certain exceptions can be handled. Further, a third algorithm for stochastic control of stopping time processes is presented. We explain why current policy evaluation methods do not work as critic to the proposed actor recursion. Simulation results from flow-control in communication networks attest to the performance advantages of all three algorithms.

关键词： finite horizon Markov decision processes reinforcement learning two timescale stochastic approximation actor-critic algorithms

来源：评论

学校读者我要写书评

暂无评论

actor-critic algorithms for Constrained Multi-agent Reinforcement Learning 19

Actor-Critic Algorithms for Constrained Multi-agent Reinforc...

引用

Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems

作者： Raghuram Bharadwaj Diddigi D. Sai Koti Reddy Prabuchandran K.J. Shalabh Bhatnagar Indian Institute of Science Bangalore India IBM Research Bangalore India Amazon-IISc Postdoctoral Fellow Bangalore India

ISBN: (纸本)9781450363099

Multi-agent reinforcement learning has gained lot of popularity primarily owing to the success of deep function approximation architectures. However, many real-life multi-agent applications often impose constraints on the joint action sequence that can be taken by the agents. In this work, we formulate such problems in the framework of constrained cooperative stochastic games. Under this setting, the goal of the agents is to obtain joint action sequence that minimizes a total cost objective criterion subject to total cost penalty/budget functional constraints. To this end, we utilize the Lagrangian formulation and propose actor-critic algorithms. Through experiments on a constrained multi-agent grid world task, we demonstrate that our algorithms converge to near-optimal joint action sequences satisfying the given constraints.

关键词： actor-critic algorithms multi-agent learning constrained reinforcement learning cooperative stochastic game

来源：评论

学校读者我要写书评

暂无评论

Bayesian policy gradient and actor-critic algorithms

The Journal of Machine Learning Research

引用

The Journal of Machine Learning Research 2016年第1期17卷

作者： Kevin Murphy Bernhard Schölkopf Mohammad Ghavamzadeh Yaakov Engel Michal Valko Google MPI for Intelligent Systems Adobe Research & INRIA and Adobe Research INRIA Lille Rafael Advanced Defence System Israel INRIA Lille France

Policy gradient methods are reinforcement learning algorithms that adapt a parameterized policy by following a performance gradient estimate. Many conventional policy gradient methods use Monte-Carlo techniques to estimate this gradient. The policy is improved by adjusting the parameters in the direction of the gradient estimate. Since Monte-Carlo methods tend to have high variance, a large number of samples is required to attain accurate estimates, resulting in slow convergence. In this paper, we first propose a Bayesian framework for policy gradient, based on modeling the policy gradient as a Gaussian process. This reduces the number of samples needed to obtain accurate gradient estimates. Moreover, estimates of the natural gradient as well as a measure of the uncertainty in the gradient estimates, namely, the gradient covariance, are provided at little extra cost. Since the proposed Bayesian framework considers system trajectories as its basic observable unit, it does not require the dynamics within trajectories to be of any particular form, and thus, can be easily extended to partially observable problems. On the downside, it cannot take advantage of the Markov property when the system is *** address this issue, we proceed to supplement our Bayesian policy gradient framework with a new actor-critic learning model in which a Bayesian class of non-parametric critics, based on Gaussian process temporal difference learning, is used. Such critics model the action-value function as a Gaussian process, allowing Bayes' rule to be used in computing the posterior distribution over action-value functions, conditioned on the observed data. Appropriate choices of the policy parameterization and of the prior covariance (kernel) between action-values allow us to obtain closed-form expressions for the posterior distribution of the gradient of the expected return with respect to the policy parameters. We perform detailed experimental comparisons of the proposed Bayesian

关键词： Bayesian inference Gaussian processes actor-critic algorithms policy gradient methods reinforcement learning

来源：评论

学校读者我要写书评

暂无评论

Policy Gradient and actor-critic Learning in Continuous Time and Space: Theory and algorithms

引用

JOURNAL OF MACHINE LEARNING RESEARCH 2022年第1期23卷 1-50页

作者： Jia, Yanwei Zhou, Xun Yu Columbia Univ Dept Ind Engn & Operat Res New York NY 10027 USA Columbia Univ Data Sci Inst New York NY 10027 USA

We study policy gradient (PG) for reinforcement learning in continuous time and space under the regularized exploratory formulation developed by Wang et al. (2020). We represent the gradient of the value function with respect to a given parameterized stochastic policy as the expected integration of an auxiliary running reward function that can be evaluated using samples and the current value function. This representation effectively turns PG into a policy evaluation (PE) problem, enabling us to apply the martingale approach recently developed by Jia and Zhou (2022a) for PE to solve our PG problem. Based on this analysis, we propose two types of actor-critic algorithms for RL, where we learn and update value functions and policies simultaneously and alternatingly. The first type is based directly on the aforementioned representation, which involves future trajectories and is offline. The second type, designed for online learning, employs the first-order condition of the policy gradient and turns it into martingale orthogonality conditions. These conditions are then incorporated using stochastic approximation when updating policies. Finally, we demonstrate the algorithms by simulations in two concrete examples.

关键词： reinforcement learning continuous time and space policy gradient policy evaluation actor-critic algorithms martingale

来源：评论

学校读者我要写书评

暂无评论

actor-critic Reinforcement Learning algorithms for Mean Field Games in Continuous Time, State and Action Spaces

引用

APPLIED MATHEMATICS AND OPTIMIZATION 2024年第3期89卷 72-72页

作者： Liang, Hong Chen, Zhiping Jing, Kaili Xi An Jiao Tong Univ Sch Math & Stat Xian 710049 Shaanxi Peoples R China Xian Int Acad Math & Math Technol Ctr Optimizat Tech & Quantitat Finance Xian 710049 Shaanxi Peoples R China Univ Ottawa Dept Math & Stat Ottawa ON K1N 6N5 Canada

This paper investigates mean field games in continuous time, state and action spaces with an infinite number of agents, where each agent aims to maximize its expected cumulative reward. Using the technique of randomized policies, we show policy evaluation and policy gradient are equivalent to the martingale conditions of a process by focusing on a representative agent. Then combined with fictitious game, we propose online and offline actor-critic algorithms for solving continuous mean field games that update the value function and policy alternatively under the given population state and action distributions. We demonstrate through two numerical experiments that our proposed algorithms can converge to the mean field equilibrium quickly and stably.

关键词： Mean field games Entropy regularization Martingale actor-critic algorithms Linear-quadratic games

来源：评论

学校读者我要写书评

暂无评论

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案：

请选择收藏分类：

通借通还

建议与咨询 留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

时间限定

文献类型

馆藏选择

核心期刊

语言

文献类型

帮助

文字说明：

检索规则说明：

检索范例：

分类表

所选分类

限定检索结果

文献类型

馆藏范围

日期分布

学科分类号

主题

机构

作者

语言

请选择保存的检索档案： 新增检索档案 确定 取消

请选择收藏分类： 新增自定义分类 确定 取消

通借通还

建议与咨询留下您的常用邮箱和电话号码，以便我们向您反馈解决方案和替代方法

请选择保存的检索档案：

请选择收藏分类：