Exponential Convergence and Stability of Howard's Policy Improvement Algorithm for Controlled Diffusions

DOI10.1137/19M1236758zbMATH Open1441.93343arXiv1812.07846WikidataQ114978697 ScholiaQ114978697MaRDI QIDQ5111071FDOQ5111071

Lukasz Szpruch, B. Kerimkulov, David Šiška

Publication date: 26 May 2020

Published in: SIAM Journal on Control and Optimization (Search for Journal in Brave)

Abstract: Optimal control problems are inherently hard to solve as the optimization must be performed simultaneously with updating the underlying system. Starting from an initial guess, Howard's policy improvement algorithm separates the step of updating the trajectory of the dynamical system from the optimization and iterations of this should converge to the optimal control. In the discrete space-time setting this is often the case and even rates of convergence are known. In the continuous space-time setting of controlled diffusion the algorithm consists of solving a linear PDE followed by maximization problem. This has been shown to converge, in some situations, however no global rate of is known. The first main contribution of this paper is to establish global rate of convergence for the policy improvement algorithm and a variant, called here the gradient iteration algorithm. The second main contribution is the proof of stability of the algorithms under perturbations to both the accuracy of the linear PDE solution and the accuracy of the maximization step. The proof technique is new in this context as it uses the theory of backward stochastic differential equations.

Full work available at URL: https://arxiv.org/abs/1812.07846

zbMATH Keywords

backward stochastic differential equationstochastic controlpolicy improvement algorithm

Mathematics Subject Classification ID

Stochastic ordinary differential equations (aspects of stochastic analysis) (60H10) Optimal stochastic control (93E20) Exponential stability (93D23)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Continuous-time stochastic control and optimization with financial applications
• Controlled Markov processes and viscosity solutions
• Some Convergence Results for Howard's Algorithm
• Markovian quadratic and superquadratic BSDEs with an unbounded terminal condition
• Infinite horizon backward stochastic differential equations and elliptic equations in Hilbert spaces.
• FUNCTIONAL EQUATIONS IN THE THEORY OF DYNAMIC PROGRAMMING. V. POSITIVITY AND QUASI-LINEARITY
• Average Optimality in Markov Control Processes via Discounted-Cost Problems and Linear Programming
• SOME NEW RESULTS IN THE THEORY OF CONTROLLED DIFFUSION PROCESSES
• On the Convergence of Policy Iteration in Stationary Dynamic Programming
• Control improvement for jump-diffusion processes with applications to finance
• The rate of convergence of finite-difference approximations for parabolic bellman equations with Lipschitz coefficients in cylindrical domains
• On the convergence of policy iteration for controlled diffusions
• Convergence Properties of Policy Iteration
• On finite-difference approximations for normalized Bellman equations
• On the policy improvement algorithm in continuous time

Cited In (13)

• Rates of convergence for the policy iteration method for mean field games systems
• Gradient flows for regularized stochastic control problems
• Exploratory LQG mean field games with entropy regularization
• A modified MSA for stochastic control problems
• Linear Convergence of a Policy Gradient Method for Some Finite Horizon Continuous Time Control Problems
• A Modified Method of Successive Approximations for Stochastic Recursive Optimal Control Problems
• A neural network-based policy iteration algorithm with global \(H^2\)-superlinear convergence for stochastic games on domains
• The modified MSA, a gradient flow and convergence
• Market based mechanisms for incentivising exchange liquidity provision
• A policy iteration method for mean field games
• Reinforcement Learning for Linear-Convex Models with Jumps via Stability Analysis of Feedback Controls
• Improved order 1/4 convergence for piecewise constant policy approximation of stochastic control problems
• Policy iteration method for time-dependent mean field games systems with non-separable Hamiltonians

This page was built for publication: Exponential Convergence and Stability of Howard's Policy Improvement Algorithm for Controlled Diffusions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5111071)