Reinforced optimal control

DOI10.4310/CMS.2022.V20.N7.A7MaRDI QIDQ2103076zbMATH OpenFDO

Authors Christian Bayer, D. Belomestny, Paul Hager, Paolo Pigato, John Schoenmakers, V. G. Spokoiny

Publication date 13 December 2022

Published in Communications in Mathematical Sciences (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2011.12382

zbMATH Keywords

Monte Carloregressionoptimal controlreinforcement learning

Mathematics Subject Classification ID

Dynamic programming in optimal control and differential games (49L20) Markov and semi-Markov decision processes (90C40) Optimal stochastic control (93E20) Least squares and related methods for stochastic control systems (93E24)

Abstract: Least squares Monte Carlo methods are a popular numerical approximation method for solving stochastic control problems. Based on dynamic programming, their key feature is the approximation of the conditional expectation of future rewards by linear least squares regression. Hence, the choice of basis functions is crucial for the accuracy of the method. Earlier work by some of us [Belomestny, Schoenmakers, Spokoiny, Zharkynbay. Commun.~Math.~Sci., 18(1):109-121, 2020](arXiv:1808.02341) proposes to reinforce the basis functions in the case of optimal stopping problems by already computed value functions for later times, thereby considerably improving the accuracy with limited additional computational cost. We extend the reinforced regression method to a general class of stochastic control problems, while considerably improving the method's efficiency, as demonstrated by substantial numerical examples as well as theoretical analysis.

Recommendations

• Optimal stopping via reinforced regression
• An efficient algorithm for stochastic optimal control problems by means of a least-squares Monte-Carlo method
• Regression methods for stochastic control problems and their convergence analysis
• Generic improvements to least squares Monte Carlo methods with applications to optimal stopping problems
• Monte Carlo methods via a dual approach for some discrete time stochastic control problems

Cited in

(7)

• A bias-corrected least-squares Monte Carlo for solving multi-period utility models
• Generic improvements to least squares Monte Carlo methods with applications to optimal stopping problems
• Optimal stopping via reinforced regression
• Relationship between least squares Monte Carlo and approximate linear programming
• Value function estimators for Feynman-Kac forward-backward SDEs in stochastic optimal control
• Premium control with reinforcement learning
• Regression methods for stochastic control problems and their convergence analysis

This page was built for publication: Reinforced optimal control

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2103076)