Natural actor-critic algorithms

DOI10.1016/j.automatica.2009.07.008zbMath1183.93130MaRDI QIDQ1049136

Shalabh Bhatnagar, Richard S. Sutton, Mohammad Ghavamzadeh, Mark Lee

Publication date: 8 January 2010

Published in: Automatica (Search for Journal in Brave)

Full work available at URL: https://doi.org/10.1016/j.automatica.2009.07.008

zbMATH Keywords

temporal difference learning; function approximation; approximate dynamic programming; natural gradient; actor-critic reinforcement learning algorithms; policy-gradient methods; two-timescale stochastic approximation

Mathematics Subject Classification ID

49L20: Dynamic programming in optimal control and differential games

60J20: Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.)

93E35: Stochastic learning and adaptive control

Related Items

An online actor-critic algorithm with function approximation for constrained Markov decision processes, Hessian matrix distribution for Bayesian policy gradient reinforcement learning, The Borkar-Meyn theorem for asynchronous stochastic approximations, An actor-critic algorithm with function approximation for discounted cost constrained Markov decision processes, The factored policy-gradient planner, Natural actor-critic algorithms, Preference-based reinforcement learning: a formal framework and a policy iteration algorithm

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Nonconvergence to unstable points in urn models and stochastic approximations
• Natural actor-critic algorithms
• Stochastic approximation methods for constrained and unconstrained systems
• Elevator group control using multiple reinforcement learning agents
• Asynchronous stochastic approximation and Q-learning
• Stochastic approximation with two time scales
• Average cost temporal-difference learning
• Simple statistical gradient-following algorithms for connectionist reinforcement learning
• Reinforcement learning based algorithms for average cost Markov decision processes
• Learning Algorithms for Markov Decision Processes with Average Cost
• Functional Approximations and Dynamic Programming
• Some Pathological Traps for Stochastic Approximation
• A Survey of Applications of Markov Decision Processes
• An analysis of temporal-difference learning with function approximation
• Perturbation realization, potentials, and sensitivity analysis of Markov processes
• OnActor-Critic Algorithms
• Simulation-based optimization of Markov reward processes
• Adaptive multivariate three-timescale stochastic approximation algorithms for simulation based optimization
• Adaptive Newton-based multivariate smoothed functional algorithms for simulation optimization
• 10.1162/1532443041827907
• Actor-Critic--Type Learning Algorithms for Markov Decision Processes
• The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning
• A Simultaneous Perturbation Stochastic Approximation-Based Actor–Critic Algorithm for Markov Decision Processes
• On the convergence of temporal-difference learning with linear function approximation