Linear Convergence of a Policy Gradient Method for Some Finite Horizon Continuous Time Control Problems

DOI10.1137/22M1492180arXiv2203.11758OpenAlexW4389301972MaRDI QIDQ6140987FDOQ6140987

Authors: C. Reisinger, Wolfgang Stockinger, Yu-Fei Zhang 

Publication date: 2 January 2024

Published in: SIAM Journal on Control and Optimization (Search for Journal in Brave)

Abstract: Despite its popularity in the reinforcement learning community, a provably convergent policy gradient method for continuous space-time control problems with nonlinear state dynamics has been elusive. This paper proposes proximal gradient algorithms for feedback controls of finite-time horizon stochastic control problems. The state dynamics are nonlinear diffusions with control-affine drift, and the cost functions are nonconvex in the state and nonsmooth in the control. The system noise can degenerate, which allows for deterministic control problems as special cases. We prove under suitable conditions that the algorithm converges linearly to a stationary point of the control problem, and is stable with respect to policy updates by approximate gradient steps. The convergence result justifies the recent reinforcement learning heuristics that adding entropy regularization or a fictitious discount factor to the optimization objective accelerates the convergence of policy gradient methods. The proof exploits careful regularity estimates of backward stochastic differential equations.

Full work available at URL: https://arxiv.org/abs/2203.11758

zbMATH Keywords

backward stochastic differential equationstochastic controlstationary pointreinforcement learninglinear convergencepolicy gradient method

Mathematics Subject Classification ID

Analysis of algorithms and problem complexity (68Q25) Optimal stochastic control (93E20) Numerical methods based on necessary conditions (49M05)

Cites Work

• A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
• Variational Analysis
• Introductory lectures on convex optimization. A basic course.
• Continuous-time stochastic control and optimization with financial applications
• Nonlinear control systems: An introduction
• A regression-based Monte Carlo method to solve backward stochastic differential equations
• Title not available (Why is that?)
• Title not available (Why is that?)
• On the convergence rate of approximation schemes for Hamilton-Jacobi-Bellman Equations
• Mean field games and mean field type control theory
• Convexity and optimization in Banach spaces.
• Reinforcement learning. An introduction
• Lectures on BSDEs, stochastic control, and stochastic differential games with financial applications
• Sufficient stochastic maximum principle for discounted control problem
• BSDEs with polynomial growth generators
• Policy gradient in continuous time
• Time discretization and Markovian iteration for coupled FBSDEs
• A numerical scheme for a mean field game in some queueing systems based on Markov chain approximation method
• Deep backward schemes for high-dimensional nonlinear PDEs
• Solving high-dimensional partial differential equations using deep learning
• Time discretization of FBSDE with polynomial growth drivers and reaction-diffusion PDEs
• Backward stochastic differential equations. From linear to fully nonlinear theory
• Maximum principle based algorithms for deep learning
• Exponential convergence and stability of Howard's policy improvement algorithm for controlled diffusions
• A neural network-based policy iteration algorithm with global \(H^2\)-superlinear convergence for stochastic games on domains
• A modified MSA for stochastic control problems
• Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon

Cited In (5)

• An explicit Milstein-type scheme for interacting particle systems and McKean-Vlasov SDEs with common noise and non-differentiable drift coefficients
• A fast iterative PDE-based algorithm for feedback controls of nonsmooth mean-field control problems
• The modified MSA, a gradient flow and convergence
• Near optimality of Lipschitz and smooth policies in controlled diffusions
• Improved order 1/4 convergence for piecewise constant policy approximation of stochastic control problems

This page was built for publication: Linear Convergence of a Policy Gradient Method for Some Finite Horizon Continuous Time Control Problems

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6140987)