Block Policy Mirror Descent

DOI10.1137/22M1480409MaRDI QIDQ6093281zbMATH OpenFDO

Authors

Publication date 6 September 2023

Published in SIAM Journal on Optimization (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2201.05756

zbMATH Keywords

block coordinate decentiteration and sample complexityMarkov decision processmirror descentpolicy gradientreinforcement learning

Mathematics Subject Classification ID

Analysis of algorithms and problem complexity (68Q25) Stochastic programming (90C15) Nonconvex programming, global optimization (90C26) Markov and semi-Markov decision processes (90C40)

Abstract: In this paper, we present a new policy gradient (PG) methods, namely the block policy mirror descent (BPMD) method for solving a class of regularized reinforcement learning (RL) problems with (strongly)-convex regularizers. Compared to the traditional PG methods with a batch update rule, which visits and updates the policy for every state, BPMD method has cheap per-iteration computation via a partial update rule that performs the policy update on a sampled state. Despite the nonconvex nature of the problem and a partial update rule, we provide a unified analysis for several sampling schemes, and show that BPMD achieves fast linear convergence to the global optimality. In particular, uniform sampling leads to comparable worst-case total computational complexity as batch PG methods. A necessary and sufficient condition for convergence with on-policy sampling is also identified. With a hybrid sampling scheme, we further show that BPMD enjoys potential instance-dependent acceleration, leading to improved dependence on the state space and consequently outperforming batch PG methods. We then extend BPMD methods to the stochastic setting, by utilizing stochastic first-order information constructed from samples. With a generative model, $ildemathcalO(leftlvertmathcalSightvertleftlvertmathcalAightvert/epsilon)$ (resp. $ildemathcalO(leftlvertmathcalSightvertleftlvertmathcalAightvert/epsilo{n}^2)$) sample complexities are established for the strongly-convex (resp. non-strongly-convex) regularizers, where $epsilon$ denotes the target accuracy. To the best of our knowledge, this is the first time that block coordinate descent methods have been developed and analyzed for policy optimization in reinforcement learning, which provides a new perspective on solving large-scale RL problems.

Recommendations

• Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes
• Approximate Newton Policy Gradient Algorithms
• Proximal gradient temporal difference learning: stable reinforcement learning with polynomial sample complexity
• Fast global convergence of natural policy gradient methods with entropy regularization
• On linear and super-linear convergence of natural policy gradient algorithm

Cites work

• Block stochastic gradient iteration for convex and nonconvex optimization
• Convergence of a block coordinate descent method for nondifferentiable minimization
• Coordinate descent algorithms
• Efficiency of coordinate descent methods on huge-scale optimization problems
• Fast global convergence of natural policy gradient methods with entropy regularization
• scientific article; zbMATH DE number 3148886 (Why is no real title available?)
• scientific article; zbMATH DE number 6508162 (Why is no real title available?)
• scientific article; zbMATH DE number 3790208 (Why is no real title available?)
• scientific article; zbMATH DE number 5685899 (Why is no real title available?)
• Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function
• Markov Decision Processes for screening and treatment of chronic diseases
• Mirror descent and nonlinear projected subgradient methods for convex optimization.
• On faster convergence of cyclic block coordinate descent-type methods for strongly convex minimization
• On linear and super-linear convergence of natural policy gradient algorithm
• On the complexity analysis of randomized block-coordinate descent methods
• On the convergence of block coordinate descent type methods
• Randomized methods for linear constraints: convergence rates and conditioning
• Reinforcement learning. An introduction
• Robust Stochastic Approximation Approach to Stochastic Programming
• Stochastic block mirror descent methods for nonsmooth and stochastic optimization

Cited in

(6)

• Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes
• Global convergence of policy gradient methods to (almost) locally optimal policies
• Softmax policy gradient methods can take exponential time to converge
• Homotopic policy mirror descent: policy convergence, algorithmic regularization, and improved sample complexity
• Policy mirror descent inherently explores action space
• Policy optimization over general state and action spaces

This page was built for publication: Block Policy Mirror Descent

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6093281)