Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

DOI10.1007/S42985-021-00102-XMaRDI QIDQ825596zbMATH OpenOpenAlexWikidataFDO

Authors Nikolas Nüsken, Lorenz Richter

Publication date 17 December 2021

Published in SN Partial Differential Equations and Applications (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2005.05409

zbMATH Keywords

deep learningrare event simulationHamilton-Jacobi-Bellman PDEsdivergences between probability measuresforward-backward SDEsiterative diffusion optimisation techniques,optimal control of diffusions

Mathematics Subject Classification ID

Hamilton-Jacobi equations (35F21) PDEs with measure (35R06)

Abstract: Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton-Jacobi-Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of $extititerativediffusionoptimisation$ techniques, in particular considering applications in importance sampling and rare event simulation, and focusing on problems without diffusion control, with linearly controlled drift and running costs that depend quadratically on the control. More generally, our methods apply to nonlinear parabolic PDEs with a certain shift invariance. The choice of an appropriate loss function being a central element in the algorithmic design, we develop a principled framework based on divergences between path measures, encompassing various existing methods. Motivated by connections to forward-backward SDEs, we propose and study the novel $extitlog-variance$ divergence, showing favourable properties of corresponding Monte Carlo estimators. The promise of the developed approach is exemplified by a range of high-dimensional and metastable numerical examples.

Recommendations

• Actor-critic method for high dimensional static Hamilton-Jacobi-Bellman partial differential equations based on neural networks
• On some neural network architectures that can represent viscosity solutions of certain high dimensional Hamilton-Jacobi partial differential equations
• scientific article; zbMATH DE number 2221967
• Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications
• Neural network architectures using min-plus algebra for solving certain high-dimensional optimal control problems and Hamilton-Jacobi PDEs
• Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
• Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis
• Path-dependent deep Galerkin method: a neural network approach to solve path-dependent partial differential equations
• Overcoming the curse of dimensionality for some Hamilton-Jacobi partial differential equations via neural network architectures
• A deep learning approach to the probabilistic numerical solution of path-dependent partial differential equations

Cites work

• scientific article; zbMATH DE number 1577097 (Why is no real title available?)
• scientific article; zbMATH DE number 3883346 (Why is no real title available?)
• scientific article; zbMATH DE number 3977666 (Why is no real title available?)
• scientific article; zbMATH DE number 3176450 (Why is no real title available?)
• scientific article; zbMATH DE number 54145 (Why is no real title available?)
• scientific article; zbMATH DE number 1325009 (Why is no real title available?)
• scientific article; zbMATH DE number 1121855 (Why is no real title available?)
• scientific article; zbMATH DE number 1500585 (Why is no real title available?)
• scientific article; zbMATH DE number 2152902 (Why is no real title available?)
• scientific article; zbMATH DE number 1909499 (Why is no real title available?)
• scientific article; zbMATH DE number 2117227 (Why is no real title available?)
• scientific article; zbMATH DE number 2117879 (Why is no real title available?)
• scientific article; zbMATH DE number 2215447 (Why is no real title available?)
• scientific article; zbMATH DE number 5187317 (Why is no real title available?)
• A Monte Carlo Method for Sensitivity Analysis and Parametric Optimization of Nonlinear Stochastic Systems
• A numerical scheme for BSDEs
• A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
• A regression-based Monte Carlo method to solve backward stochastic differential equations
• A stochastic control approach to reciprocal diffusion processes
• A variational representation for certain functionals of Brownian motion
• Adapted solution of a backward stochastic differential equation
• Adaptive importance sampling for control and inference
• Adaptive importance sampling with forward-backward stochastic differential equations
• Adaptive sampling of large deviations
• An introduction to stochastic control theory, path integrals and reinforcement learning
• Applications of the cross-entropy method to importance sampling and optimal control of diffusions
• Approximation by superpositions of a sigmoidal function
• Backward stochastic differential equations and partial differential equations with quadratic growth.
• Book review of: L. Gawarecki and V. Mandrekar, Stochastic differential equations in infinite dimensions with applications to stochastic partial differential equations
• Brownian motion in a field of force and the diffusion model of chemical reactions
• Conditional brownian motion and the boundary limits of harmonic functions
• Conditioned stochastic differential equations: theory, examples and application to finance.
• Connections between stochastic control and dynamic games
• Continuous-time stochastic control and optimization with financial applications
• Controlled Markov processes and viscosity solutions
• Controlled sequential Monte Carlo
• Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games. I: The ergodic case
• Convergence rates for optimised adaptive importance samplers
• Convergent Difference Schemes for Degenerate Elliptic and Parabolic Equations: Hamilton--Jacobi Equations and Free Boundary Problems
• Data assimilation: the Schrödinger perspective
• Deep backward schemes for high-dimensional nonlinear PDEs
• Deep learning
• Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Deep optimal stopping
• Deep relaxation: partial differential equations for optimizing deep neural networks
• Explicit solution of relative entropy weighted control
• Finite difference methods for mean field games
• Free energy computations. A mathematical perspective
• Importance Sampling, Large Deviations, and Differential Games
• Importance sampling in path space for diffusion processes with slow-fast variables
• Introduction to rare event simulation.
• Kramers law: validity, derivations and generalisations
• Large deviations for stochastic processes.
• Lectures on BSDEs, stochastic control, and stochastic differential games with financial applications
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Machine learning for semi linear PDEs
• Markov state models for rare events in molecular dynamics
• Metastability and Markov state models in molecular dynamics. Modeling, analysis, algorithmic approaches
• Metastability, conformation dynamics, and transition pathways in complex systems
• Monte-Carlo methods and stochastic processes. From linear to non-linear
• Multilayer feedforward networks are universal approximators
• Non-convergence of stochastic gradient descent in the training of deep neural networks
• Nonequilibrium Markov processes conditioned on large deviations
• Numerical probability. An introduction with applications to finance
• On Divergences and Informations in Statistics and Information Theory
• Optimal Transport
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Optimal control as a graphical model inference problem
• Optimal control of multiscale systems using reduced-order models
• Optimal stochastic control, stochastic target problems, and backward SDE.
• Overcoming the curse of dimensionality in the approximative pricing of financial derivatives with default risks
• Overcoming the curse of dimensionality in the numerical approximation of Allen-Cahn partial differential equations via truncated full-history recursive multilevel Picard approximations
• Overcoming the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Probabilistic theory of mean field games with applications I. Mean field FBSDEs, control, and games
• Probability theory. A comprehensive course
• Proof of the convergence of the successive approximation algorithm for numerically solving the Hamilton-Jacobi-Bellman equation
• Proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• Sensitivity Analysis Using Itô--Malliavin Calculus and Martingales, and Application to Stochastic Optimal Control
• Solving high-dimensional optimal stopping problems using deep learning
• Solving high-dimensional partial differential equations using deep learning
• Solving the Kolmogorov PDE by means of deep learning
• Stochastic Control Theory
• Stochastic processes and applications. Diffusion processes, the Fokker-Planck and Langevin equations
• The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems
• Transformation of measure on Wiener space
• Variational Monte Carlo -- bridging concepts of machine learning and high-dimensional partial differential equations
• Variational approach to rare event simulation using least-squares regression

Cited in

(29)

• Numerical solutions of sea turtle population dynamics model by using restarting strategy of PINN-Adam
• Importance sampling for the empirical measure of weakly interacting diffusions
• Learning-based importance sampling via stochastic optimal control for stochastic reaction networks
• Neural parametric Fokker-Planck equation
• State-dependent Riccati equation feedback stabilization for nonlinear PDEs
• Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
• An overview on deep learning-based approximation methods for partial differential equations
• Actor-critic method for high dimensional static Hamilton-Jacobi-Bellman partial differential equations based on neural networks
• Connecting stochastic optimal control and reinforcement learning
• A neural network approach for stochastic optimal control
• Overcoming the timescale barrier in molecular dynamics: Transfer operators, variational principles and machine learning
• Approximation error analysis of some deep backward schemes for nonlinear PDEs
• Mobility Estimation for Langevin Dynamics Using Control Variates
• Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing
• Approximative policy iteration for exit time feedback control problems driven by stochastic differential equations using tensor train format
• Stein variational gradient descent: many-particle and long-time asymptotics
• Neural network architectures using min-plus algebra for solving certain high-dimensional optimal control problems and Hamilton-Jacobi PDEs
• Hamilton-Jacobi equations and mathematical morphology in pseudo-Riemannian manifolds
• Learning Koopman eigenfunctions of stochastic diffusions with optimal importance sampling and ISOKANN
• A Neural Network Approach to High-Dimensional Optimal Switching Problems with Jumps in Energy Markets
• Neural Control of Parametric Solutions for High-Dimensional Evolution PDEs
• Double-loop importance sampling for McKean-Vlasov stochastic differential equation
• Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning
• Approximating optimal feedback controllers of finite horizon control problems using hierarchical tensor formats
• Numerical methods for backward stochastic differential equations: a survey
• Approximation of optimal feedback controls for stochastic reaction-diffusion equations
• Bayesian learning via neural Schrödinger-Föllmer flows
• Error bounds for model reduction of feedback-controlled linear stochastic dynamics on Hilbert spaces
• Extensions of the deep Galerkin method

This page was built for publication: Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q825596)