Fast global convergence of natural policy gradient methods with entropy regularization
From MaRDI portal
Abstract: Natural policy gradient (NPG) methods are among the most widely used policy optimization algorithms in contemporary reinforcement learning. This class of methods is often applied in conjunction with entropy regularization -- an algorithmic scheme that encourages exploration -- and is closely related to soft policy iteration and trust region policy optimization. Despite the empirical success, the theoretical underpinnings for NPG methods remain limited even for the tabular setting. This paper develops convergence guarantees for entropy-regularized NPG methods under softmax parameterization, focusing on discounted Markov decision processes (MDPs). Assuming access to exact policy evaluation, we demonstrate that the algorithm converges linearly -- or even quadratically once it enters a local region around the optimal policy -- when computing optimal value functions of the regularized MDP. Moreover, the algorithm is provably stable vis-`a-vis inexactness of policy evaluation. Our convergence results accommodate a wide range of learning rates, and shed light upon the role of entropy regularization in enabling fast convergence.
Recommendations
- Approximate Newton Policy Gradient Algorithms
- On linear and super-linear convergence of natural policy gradient algorithm
- Softmax policy gradient methods can take exponential time to converge
- Entropy Regularization for Mean Field Games with Learning
- Global convergence of policy gradient methods to (almost) locally optimal policies
Cites work
- Dynamic programming and optimal control. Vol. 1.
- Global convergence of policy gradient methods to (almost) locally optimal policies
- scientific article; zbMATH DE number 3790208 (Why is no real title available?)
- Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model
- Natural actor-critic algorithms
- On the Theory of Dynamic Programming
- Online Markov decision processes
- Primal-dual subgradient methods for convex problems
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
Cited in
(33)- Compatible natural gradient policy search
- On linear and super-linear convergence of natural policy gradient algorithm
- Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes
- On the theory of policy gradient methods: optimality, approximation, and distribution shift
- Global convergence of policy gradient methods to (almost) locally optimal policies
- Policy optimization for \(\mathcal{H}_2\) linear control with \(\mathcal{H}_\infty\) robustness guarantee: implicit regularization and global convergence
- Entropy Regularization for Mean Field Games with Learning
- Approximate Newton Policy Gradient Algorithms
- Block Policy Mirror Descent
- Softmax policy gradient methods can take exponential time to converge
- Geometry and convergence of natural policy gradient methods
- Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
- Accelerating Primal-Dual Methods for Regularized Markov Decision Processes
- Entropy regularization methods for parameter space exploration
- Homotopic policy mirror descent: policy convergence, algorithmic regularization, and improved sample complexity
- Recent developments in machine learning methods for stochastic control and games
- Global convergence of natural policy gradient with Hessian-aided momentum variance reduction
- Global optimality guarantees for policy gradient methods
- Policy mirror descent inherently explores action space
- Fast policy learning for linear-quadratic control with entropy regularization
- On the convergence of projected policy gradient for any constant step sizes
- Score-aware policy-gradient and performance guarantees using local Lyapunov stability
- Regularized minimax-V learning for solving randomly terminating two-player zero-sum Markov games
- All-time safety and sample-efficient meta update for online safe meta reinforcement learning under Markov task transition
- Sublinear regret for a class of continuous-time linear-quadratic reinforcement learning problems
- Policy gradient converges to the globally optimal policy for nearly linear-quadratic regulators
- An approximate policy iteration viewpoint of actor-critic algorithms
- Fast computation of optimal transport via entropy-regularized extragradient methods
- Fisher-Rao gradient flows of linear programs and state-action natural policy gradients
- Optimal rates of convergence for entropy regularization in discounted Markov decision processes
- Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs
- Policy optimization over general state and action spaces
- A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spaces
This page was built for publication: Fast global convergence of natural policy gradient methods with entropy regularization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5106383)
