A multilevel approach for stochastic nonlinear optimal control

DOI10.1080/00207179.2020.1849805MaRDI QIDQ5863708zbMATH OpenOpenAlexFDO

Authors Ajay Jasra, Jeremy Heng, Yaxian Xu, Adrian N. Bishop

Publication date 3 June 2022

Published in International Journal of Control (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1901.05583

zbMATH Keywords

Markov chain Monte Carlomultilevel Monte Carlooptimal controlsequential Monte Carlo

Mathematics Subject Classification ID

Linear-quadratic optimal control problems (49N10) Nonlinear systems in control theory (93C10) Optimal stochastic control (93E20)

Abstract: We consider a class of finite time horizon nonlinear stochastic optimal control problem, where the control acts additively on the dynamics and the control cost is quadratic. This framework is flexible and has found applications in many domains. Although the optimal control admits a path integral representation for this class of control problems, efficient computation of the associated path integrals remains a challenging Monte Carlo task. The focus of this article is to propose a new Monte Carlo approach that significantly improves upon existing methodology. Our proposed methodology first tackles the issue of exponential growth in variance with the time horizon by casting optimal control estimation as a smoothing problem for a state space model associated with the control problem, and applying smoothing algorithms based on particle Markov chain Monte Carlo. To further reduce computational cost, we then develop a multilevel Monte Carlo method which allows us to obtain an estimator of the optimal control with $mathcalO(epsilo{n}^2)$ mean squared error with a computational cost of $mathcalO(epsilo{n}^{-2}log(epsilon{)}^2)$. In contrast, a computational cost of $mathcalO(epsilo{n}^{-3})$ is required for existing methodology to achieve the same mean squared error. Our approach is illustrated on two numerical examples, which validate our theory.

Recommendations

• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Pathwise Stochastic Optimal Control
• Iterative path integral approach to nonlinear stochastic optimal control under compound Poisson noise
• An efficient algorithm for stochastic optimal control problems by means of a least-squares Monte-Carlo method
• Path integrals and symmetry breaking for optimal control theory

Cites work

• A generalized path integral control approach to reinforcement learning
• A probabilistic numerical method for fully nonlinear parabolic PDEs
• Adaptive importance sampling for control and inference
• Bayesian static parameter estimation for partially observed diffusions via multilevel Monte Carlo
• Concepts and methods for discrete and continuous time control under uncertainty
• Continuous-time stochastic control and optimization with financial applications
• Control of a Solution of a Stochastic Integral Equation
• Controlled Markov processes and viscosity solutions
• Controlled sequential Monte Carlo
• Dynamics of drug resistance: optimal control of an infectious disease
• scientific article; zbMATH DE number 16922 (Why is no real title available?)
• scientific article; zbMATH DE number 3438157 (Why is no real title available?)
• scientific article; zbMATH DE number 2000348 (Why is no real title available?)
• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Multilevel Monte Carlo Path Simulation
• Multilevel Particle Filters
• Nonlinear stochastic receding horizon control: stability, robustness and Monte Carlo methods for control approximation
• Optimal Control and Nonlinear Filtering for Nondegenerate Diffusion Processes
• Optimal control as a graphical model inference problem
• Optimal Control of the Vidale-Wolfe Advertising Model
• Optimal control theory. Applications to management science and economics
• Particle Markov Chain Monte Carlo Methods
• Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother
• Path integrals and symmetry breaking for optimal control theory
• Sequential Monte Carlo methods for diffusion processes
• Stochastic control theory and operational research
• Unbiased inference for discretely observed hidden Markov model diffusions
• Uniform ergodicity of the iterated conditional SMC and geometric ergodicity of particle Gibbs samplers
• Vaccination control in a stochastic SVIR epidemic model

Cited in

(13)

• An efficient Monte Carlo method for optimal control problems with uncertainty
• Multilevel techniques for the solution of HJB minimum-time control problems
• Moving least-squares approximations for linearly-solvable stochastic optimal control problems
• EP for efficient stochastic control with obstacles
• Path integrals and symmetry breaking for optimal control theory
• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Iterative path integral approach to nonlinear stochastic optimal control under compound Poisson noise
• Optimal Control and Stochastic Parameter Estimation
• Bayesian parameter inference for partially observed stochastic volterra equations
• Modeling of measurement error in financial returns data
• Bayesian parameter estimation for partially observed McKean-Vlasov diffusions using multilevel Markov chain Monte Carlo
• Multilevel Monte Carlo for a class of partially observed processes in neuroscience
• Bayesian inference for non-synchronously observed diffusions

This page was built for publication: A multilevel approach for stochastic nonlinear optimal control

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5863708)