Geometry and convergence of natural policy gradient methods
From MaRDI portal
(Redirected from Publication:6138809)
Abstract: We study the convergence of several natural policy gradient (NPG) methods in infinite-horizon discounted Markov decision processes with regular policy parametrizations. For a variety of NPGs and reward functions we show that the trajectories in state-action space are solutions of gradient flows with respect to Hessian geometries, based on which we obtain global convergence guarantees and convergence rates. In particular, we show linear convergence for unregularized and regularized NPG flows with the metrics proposed by Kakade and Morimura and co-authors by observing that these arise from the Hessian geometries of conditional entropy and entropy respectively. Further, we obtain sublinear convergence rates for Hessian geometries arising from other convex functions like log-barriers. Finally, we interpret the discrete-time NPG methods with regularized rewards as inexact Newton methods if the NPG is defined with respect to the Hessian geometry of the regularizer. This yields local quadratic convergence rates of these methods for step size equal to the penalization strength.
Recommendations
- Fast global convergence of natural policy gradient methods with entropy regularization
- On linear and super-linear convergence of natural policy gradient algorithm
- Global convergence of natural policy gradient with Hessian-aided momentum variance reduction
- Approximate Newton Policy Gradient Algorithms
- Global convergence of policy gradient methods to (almost) locally optimal policies
Cites work
- An Extended Cencov Characterization of the Information Metric
- Approximate Newton Policy Gradient Algorithms
- Axiomatic Geometry of Conditional Models
- Derivatives of logarithmic stationary distributions for policy gradient reinforcement learning
- Efficient Natural Gradient Descent Methods for Large-Scale PDE-Based Optimization Problems
- Fast global convergence of natural policy gradient methods with entropy regularization
- Finite state Markovian decision processes
- Global convergence of policy gradient methods to (almost) locally optimal policies
- Hessian informed mirror descent
- Hessian Riemannian Gradient Flows in Convex Programming
- scientific article; zbMATH DE number 3761167 (Why is no real title available?)
- scientific article; zbMATH DE number 1753152 (Why is no real title available?)
- scientific article; zbMATH DE number 2107836 (Why is no real title available?)
- scientific article; zbMATH DE number 7306852 (Why is no real title available?)
- Inexact Newton Methods
- Information geometry
- Information geometry and its applications
- Invariance properties of the natural gradient in overparametrised systems
- Natural gradient via optimal transport
- On the Fisher metric of conditional probability polytopes
- On the theory of policy gradient methods: optimality, approximation, and distribution shift
- OnActor-Critic Algorithms
- Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
- Reinforcement learning. An introduction
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
- Simulation-based optimization of Markov reward processes
- Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory
- The geometry of Hessian structures
- Wasserstein Riemannian geometry of Gaussian densities
Cited in
(4)- Global convergence of natural policy gradient with Hessian-aided momentum variance reduction
- DRM revisited: a complete error analysis
- Fisher-Rao gradient flows of linear programs and state-action natural policy gradients
- Optimal rates of convergence for entropy regularization in discounted Markov decision processes
This page was built for publication: Geometry and convergence of natural policy gradient methods
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6138809)
