Fast global convergence of natural policy gradient methods with entropy regularization
From MaRDI portal
Abstract: Natural policy gradient (NPG) methods are among the most widely used policy optimization algorithms in contemporary reinforcement learning. This class of methods is often applied in conjunction with entropy regularization -- an algorithmic scheme that encourages exploration -- and is closely related to soft policy iteration and trust region policy optimization. Despite the empirical success, the theoretical underpinnings for NPG methods remain limited even for the tabular setting. This paper develops convergence guarantees for entropy-regularized NPG methods under softmax parameterization, focusing on discounted Markov decision processes (MDPs). Assuming access to exact policy evaluation, we demonstrate that the algorithm converges linearly -- or even quadratically once it enters a local region around the optimal policy -- when computing optimal value functions of the regularized MDP. Moreover, the algorithm is provably stable vis-`a-vis inexactness of policy evaluation. Our convergence results accommodate a wide range of learning rates, and shed light upon the role of entropy regularization in enabling fast convergence.
Recommendations
- Approximate Newton Policy Gradient Algorithms
- On linear and super-linear convergence of natural policy gradient algorithm
- Softmax policy gradient methods can take exponential time to converge
- Entropy Regularization for Mean Field Games with Learning
- Global convergence of policy gradient methods to (almost) locally optimal policies
Cites work
- scientific article; zbMATH DE number 3790208 (Why is no real title available?)
- Dynamic programming and optimal control. Vol. 1.
- Global convergence of policy gradient methods to (almost) locally optimal policies
- Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model
- Natural actor-critic algorithms
- On the Theory of Dynamic Programming
- Online Markov decision processes
- Primal-dual subgradient methods for convex problems
- Simple statistical gradient-following algorithms for connectionist reinforcement learning
Cited in
(29)- An approximate policy iteration viewpoint of actor-critic algorithms
- Geometry and convergence of natural policy gradient methods
- Fast computation of optimal transport via entropy-regularized extragradient methods
- Fisher-Rao gradient flows of linear programs and state-action natural policy gradients
- Approximate Newton Policy Gradient Algorithms
- Homotopic policy mirror descent: policy convergence, algorithmic regularization, and improved sample complexity
- Entropy Regularization for Mean Field Games with Learning
- Recent developments in machine learning methods for stochastic control and games
- Optimal rates of convergence for entropy regularization in discounted Markov decision processes
- Softmax policy gradient methods can take exponential time to converge
- Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergence
- Global convergence of natural policy gradient with Hessian-aided momentum variance reduction
- On linear and super-linear convergence of natural policy gradient algorithm
- Accelerating Primal-Dual Methods for Regularized Markov Decision Processes
- Fast policy learning for linear-quadratic control with entropy regularization
- Block Policy Mirror Descent
- Entropy regularization methods for parameter space exploration
- On the convergence of projected policy gradient for any constant step sizes
- Score-aware policy-gradient and performance guarantees using local Lyapunov stability
- Global optimality guarantees for policy gradient methods
- Regularized minimax-V learning for solving randomly terminating two-player zero-sum Markov games
- Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes
- All-time safety and sample-efficient meta update for online safe meta reinforcement learning under Markov task transition
- Policy mirror descent inherently explores action space
- Compatible natural gradient policy search
- Sublinear regret for a class of continuous-time linear-quadratic reinforcement learning problems
- Global convergence of policy gradient methods to (almost) locally optimal policies
- Policy optimization for \(\mathcal{H}_2\) linear control with \(\mathcal{H}_\infty\) robustness guarantee: implicit regularization and global convergence
- Policy gradient converges to the globally optimal policy for nearly linear-quadratic regulators
This page was built for publication: Fast global convergence of natural policy gradient methods with entropy regularization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5106383)
