Natural actor-critic algorithms

DOI10.1016/j.automatica.2009.07.008zbMath1183.93130OpenAlexW2094387729MaRDI QIDQ1049136

Mohammad Ghavamzadeh, Mark Lee, Richard S. Sutton, Shalabh Bhatnagar

Publication date: 8 January 2010

Published in: Automatica (Search for Journal in Brave)

Full work available at URL: https://doi.org/10.1016/j.automatica.2009.07.008

zbMATH Keywords

temporal difference learningfunction approximationapproximate dynamic programmingnatural gradientactor-critic reinforcement learning algorithmspolicy-gradient methodstwo-timescale stochastic approximation

Mathematics Subject Classification ID

Dynamic programming in optimal control and differential games (49L20) Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) (60J20) Stochastic learning and adaptive control (93E35)

Related Items

An incremental off-policy search in a model-free Markov decision process using a single sample path ⋮ A constrained optimization perspective on actor-critic algorithms and application to network routing ⋮ The factored policy-gradient planner ⋮ Temporal concatenation for Markov decision processes ⋮ A unified DC programming framework and efficient DCA based approaches for large scale batch reinforcement learning ⋮ Multiscale Q-learning with linear function approximation ⋮ Unnamed Item ⋮ A stability criterion for two timescale stochastic approximation schemes ⋮ On linear and super-linear convergence of natural policy gradient algorithm ⋮ An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes ⋮ Risk-Sensitive Reinforcement Learning via Policy Gradient Search ⋮ Variance-constrained actor-critic algorithms for discounted and average reward MDPs ⋮ Multi-agent off-policy actor-critic algorithm for distributed multi-task reinforcement learning ⋮ Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization ⋮ Unnamed Item ⋮ On the sample complexity of actor-critic method for reinforcement learning with function approximation ⋮ Multi-agent natural actor-critic reinforcement learning algorithms ⋮ Reinforcement learning algorithms with function approximation: recent advances and applications ⋮ Preference-based reinforcement learning: a formal framework and a policy iteration algorithm ⋮ Dynamics and risk sharing in groups of selfish individuals ⋮ An online actor-critic algorithm with function approximation for constrained Markov decision processes ⋮ Unnamed Item ⋮ Nonconvex Policy Search Using Variational Inequalities ⋮ Autonomous reinforcement learning with experience replay ⋮ Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies ⋮ Learning and control of exploration primitives ⋮ Deep Reinforcement Learning: A State-of-the-Art Walkthrough ⋮ Parameterized Markov decision process and its application to service rate control ⋮ Risk-Constrained Reinforcement Learning with Percentile Risk Criteria ⋮ Hessian matrix distribution for Bayesian policy gradient reinforcement learning ⋮ The Borkar-Meyn theorem for asynchronous stochastic approximations ⋮ Real-time reinforcement learning by sequential actor-critics and experience replay ⋮ Adaptive critic design with graph Laplacian for online learning control of nonlinear systems ⋮ Unnamed Item ⋮ Natural actor-critic algorithms ⋮ Multi-agent reinforcement learning: a selective overview of theories and algorithms ⋮ Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks ⋮ Actor-Critic Algorithms with Online Feature Adaptation

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Nonconvergence to unstable points in urn models and stochastic approximations
• Natural actor-critic algorithms
• Stochastic approximation methods for constrained and unconstrained systems
• Elevator group control using multiple reinforcement learning agents
• Asynchronous stochastic approximation and Q-learning
• Stochastic approximation with two time scales
• Average cost temporal-difference learning
• Simple statistical gradient-following algorithms for connectionist reinforcement learning
• Reinforcement learning based algorithms for average cost Markov decision processes
• Learning Algorithms for Markov Decision Processes with Average Cost
• Functional Approximations and Dynamic Programming
• Some Pathological Traps for Stochastic Approximation
• A Survey of Applications of Markov Decision Processes
• An analysis of temporal-difference learning with function approximation
• Perturbation realization, potentials, and sensitivity analysis of Markov processes
• OnActor-Critic Algorithms
• Simulation-based optimization of Markov reward processes
• Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization
• Adaptive Newton-based multivariate smoothed functional algorithms for simulation optimization
• 10.1162/1532443041827907
• Actor-Critic--Type Learning Algorithms for Markov Decision Processes
• The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
• A Simultaneous Perturbation Stochastic Approximation-Based Actor–Critic Algorithm for Markov Decision Processes
• On the convergence of temporal-difference learning with linear function approximation