The Modified MSA, a Gradient Flow and Convergence

MaRDI QIDQ6420242zbMATH OpenFDO

Authors Deven Sethi, David Šiška

Publication date 12 December 2022

Mathematics Subject Classification ID

Dynamical systems in optimization and economics (37N40) Applications of stochastic analysis (to PDEs, etc.) (60H30) Numerical methods for mathematical programming, optimization and variational techniques (65K99) Optimal stochastic control (93E20)

Abstract: The modified Method of Successive Approximations (MSA) is an iterative scheme for approximating solutions to stochastic control problems in continuous time based on Pontryagin Optimality Principle which, starting with an initial open loop control, solves the forward equation, the backward adjoint equation and then performs a static minimization step. We observe that this is an implicit Euler scheme for a gradient flow system. We prove that appropriate interpolations of the iterates of the modified MSA converge to a gradient flow. We then study the convergence of this gradient flow as time goes to infinity. In the general (non-convex) case we prove that the gradient term itself converges to zero. This is a consequence of an energy identity we establish which shows that the optimization objective decreases along the gradient flow. Moreover, in the convex case, when Pontryagin Optimality Principle provides a sufficient condition for optimality, we prove that the optimization objective converges at rate $frac1S$ to its optimal value. The main technical difficulties lie in obtaining appropriate properties of the Hamiltonian (growth, continuity). These are obtained by utilising the theory of Bounded Mean Oscillation (BMO) martingales required for estimates on the adjoint Backward Stochastic Differential Equation (BSDE).

This page was built for publication: The Modified MSA, a Gradient Flow and Convergence

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6420242)