A multilevel approach for stochastic nonlinear optimal control

DOI10.1080/00207179.2020.1849805zbMATH Open1492.93201arXiv1901.05583OpenAlexW3106433142MaRDI QIDQ5863708FDOQ5863708

Authors: Ajay Jasra, Jeremy Heng, Yaxian Xu, Adrian N. Bishop 

Publication date: 3 June 2022

Published in: International Journal of Control (Search for Journal in Brave)

Abstract: We consider a class of finite time horizon nonlinear stochastic optimal control problem, where the control acts additively on the dynamics and the control cost is quadratic. This framework is flexible and has found applications in many domains. Although the optimal control admits a path integral representation for this class of control problems, efficient computation of the associated path integrals remains a challenging Monte Carlo task. The focus of this article is to propose a new Monte Carlo approach that significantly improves upon existing methodology. Our proposed methodology first tackles the issue of exponential growth in variance with the time horizon by casting optimal control estimation as a smoothing problem for a state space model associated with the control problem, and applying smoothing algorithms based on particle Markov chain Monte Carlo. To further reduce computational cost, we then develop a multilevel Monte Carlo method which allows us to obtain an estimator of the optimal control with $mathcalO(epsilo{n}^2)$ mean squared error with a computational cost of $mathcalO(epsilo{n}^{-2}log(epsilon{)}^2)$. In contrast, a computational cost of $mathcalO(epsilo{n}^{-3})$ is required for existing methodology to achieve the same mean squared error. Our approach is illustrated on two numerical examples, which validate our theory.

Full work available at URL: https://arxiv.org/abs/1901.05583

Recommendations

• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Pathwise Stochastic Optimal Control
• Iterative path integral approach to nonlinear stochastic optimal control under compound Poisson noise
• An efficient algorithm for stochastic optimal control problems by means of a least-squares Monte-Carlo method
• Path integrals and symmetry breaking for optimal control theory

zbMATH Keywords

Markov chain Monte Carlosequential Monte Carlooptimal controlmultilevel Monte Carlo

Mathematics Subject Classification ID

Linear-quadratic optimal control problems (49N10) Nonlinear systems in control theory (93C10) Optimal stochastic control (93E20)

Cites Work

• Particle Markov Chain Monte Carlo Methods
• Title not available (Why is that?)
• Continuous-time stochastic control and optimization with financial applications
• Concepts and methods for discrete and continuous time control under uncertainty
• Controlled Markov processes and viscosity solutions
• Multilevel Monte Carlo Path Simulation
• Sequential Monte Carlo methods for diffusion processes
• Control of a Solution of a Stochastic Integral Equation
• A probabilistic numerical method for fully nonlinear parabolic PDEs
• Title not available (Why is that?)
• Title not available (Why is that?)
• A generalized path integral control approach to reinforcement learning
• Adaptive importance sampling for control and inference
• Optimal Control and Nonlinear Filtering for Nondegenerate Diffusion Processes
• Optimal control as a graphical model inference problem
• Vaccination control in a stochastic SVIR epidemic model
• Optimal Control of the Vidale-Wolfe Advertising Model
• Uniform ergodicity of the iterated conditional SMC and geometric ergodicity of particle Gibbs samplers
• Path integrals and symmetry breaking for optimal control theory
• Bayesian static parameter estimation for partially observed diffusions via multilevel Monte Carlo
• Multilevel Particle Filters
• Stochastic control theory and operational research
• Optimal control theory. Applications to management science and economics
• Controlled sequential Monte Carlo
• Dynamics of drug resistance: optimal control of an infectious disease
• Particle Smoothing for Hidden Diffusion Processes: Adaptive Path Integral Smoother
• Unbiased inference for discretely observed hidden Markov model diffusions
• Nonlinear stochastic receding horizon control: stability, robustness and Monte Carlo methods for control approximation
• Learning effective state-feedback controllers through efficient multilevel importance samplers

Cited In (9)

• Path integrals and symmetry breaking for optimal control theory
• Moving least-squares approximations for linearly-solvable stochastic optimal control problems
• Learning effective state-feedback controllers through efficient multilevel importance samplers
• Multilevel techniques for the solution of HJB minimum-time control problems
• Optimal Control and Stochastic Parameter Estimation
• Iterative path integral approach to nonlinear stochastic optimal control under compound Poisson noise
• EP for efficient stochastic control with obstacles
• An efficient Monte Carlo method for optimal control problems with uncertainty
• Bayesian parameter inference for partially observed stochastic volterra equations

This page was built for publication: A multilevel approach for stochastic nonlinear optimal control

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5863708)