Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

DOI10.1007/S42985-021-00102-XzbMATH Open1480.35101arXiv2005.05409OpenAlexW3180106877WikidataQ114216502 ScholiaQ114216502MaRDI QIDQ825596FDOQ825596

Authors: Nikolas Nüsken, Lorenz Richter 

Publication date: 17 December 2021

Published in: SN Partial Differential Equations and Applications (Search for Journal in Brave)

Abstract: Optimal control of diffusion processes is intimately connected to the problem of solving certain Hamilton-Jacobi-Bellman equations. Building on recent machine learning inspired approaches towards high-dimensional PDEs, we investigate the potential of $extititerativediffusionoptimisation$ techniques, in particular considering applications in importance sampling and rare event simulation, and focusing on problems without diffusion control, with linearly controlled drift and running costs that depend quadratically on the control. More generally, our methods apply to nonlinear parabolic PDEs with a certain shift invariance. The choice of an appropriate loss function being a central element in the algorithmic design, we develop a principled framework based on divergences between path measures, encompassing various existing methods. Motivated by connections to forward-backward SDEs, we propose and study the novel $extitlog-variance$ divergence, showing favourable properties of corresponding Monte Carlo estimators. The promise of the developed approach is exemplified by a range of high-dimensional and metastable numerical examples.

Full work available at URL: https://arxiv.org/abs/2005.05409

Recommendations

• Actor-critic method for high dimensional static Hamilton-Jacobi-Bellman partial differential equations based on neural networks
• On some neural network architectures that can represent viscosity solutions of certain high dimensional Hamilton-Jacobi partial differential equations
• scientific article; zbMATH DE number 2221967
• Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications
• Neural network architectures using min-plus algebra for solving certain high-dimensional optimal control problems and Hamilton-Jacobi PDEs
• Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
• Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis
• Path-dependent deep Galerkin method: a neural network approach to solve path-dependent partial differential equations
• Overcoming the curse of dimensionality for some Hamilton-Jacobi partial differential equations via neural network architectures
• A deep learning approach to the probabilistic numerical solution of path-dependent partial differential equations

zbMATH Keywords

deep learningrare event simulationHamilton-Jacobi-Bellman PDEsdivergences between probability measuresforward-backward SDEsiterative diffusion optimisation techniques,optimal control of diffusions

Mathematics Subject Classification ID

Hamilton-Jacobi equations (35F21) PDEs with measure (35R06)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Optimal Transport
• Deep learning
• Title not available (Why is that?)
• Continuous-time stochastic control and optimization with financial applications
• A variational representation for certain functionals of Brownian motion
• A numerical scheme for BSDEs
• Backward stochastic differential equations and partial differential equations with quadratic growth.
• Controlled Markov processes and viscosity solutions
• A regression-based Monte Carlo method to solve backward stochastic differential equations
• Title not available (Why is that?)
• Finite difference methods for mean field games
• Large deviations for stochastic processes.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Adapted solution of a backward stochastic differential equation
• Kramers law: validity, derivations and generalisations
• Title not available (Why is that?)
• Conditional brownian motion and the boundary limits of harmonic functions
• On Divergences and Informations in Statistics and Information Theory
• Optimal stochastic control, stochastic target problems, and backward SDE.
• Title not available (Why is that?)
• Brownian motion in a field of force and the diffusion model of chemical reactions
• Book review of: L. Gawarecki and V. Mandrekar, Stochastic differential equations in infinite dimensions with applications to stochastic partial differential equations
• Multilayer feedforward networks are universal approximators
• Introduction to rare event simulation.
• Approximation by superpositions of a sigmoidal function
• Connections between stochastic control and dynamic games
• Convergent Difference Schemes for Degenerate Elliptic and Parabolic Equations: Hamilton--Jacobi Equations and Free Boundary Problems
• A stochastic control approach to reciprocal diffusion processes
• Adaptive importance sampling for control and inference
• Optimal control as a graphical model inference problem
• Importance Sampling, Large Deviations, and Differential Games
• Applications of the cross-entropy method to importance sampling and optimal control of diffusions
• Markov state models for rare events in molecular dynamics
• Free energy computations. A mathematical perspective
• Stochastic processes and applications. Diffusion processes, the Fokker-Planck and Langevin equations
• Metastability and Markov state models in molecular dynamics. Modeling, analysis, algorithmic approaches
• Optimal control of multiscale systems using reduced-order models
• Title not available (Why is that?)
• Conditioned stochastic differential equations: theory, examples and application to finance.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Probability theory. A comprehensive course
• Title not available (Why is that?)
• Lectures on BSDEs, stochastic control, and stochastic differential games with financial applications
• Sensitivity Analysis Using Itô--Malliavin Calculus and Martingales, and Application to Stochastic Optimal Control
• Explicit solution of relative entropy weighted control
• Nonequilibrium Markov processes conditioned on large deviations
• Stochastic Control Theory
• Probabilistic theory of mean field games with applications I. Mean field FBSDEs, control, and games
• An introduction to stochastic control theory, path integrals and reinforcement learning
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Deep backward schemes for high-dimensional nonlinear PDEs
• Solving high-dimensional partial differential equations using deep learning
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Importance sampling in path space for diffusion processes with slow-fast variables
• A Monte Carlo Method for Sensitivity Analysis and Parametric Optimization of Nonlinear Stochastic Systems
• The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
• Proof of the convergence of the successive approximation algorithm for numerically solving the Hamilton-Jacobi-Bellman equation
• A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
• Machine learning for semi linear PDEs
• Metastability, conformation dynamics, and transition pathways in complex systems
• Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
• Numerical probability. An introduction with applications to finance
• Adaptive sampling of large deviations
• Overcoming the curse of dimensionality in the numerical approximation of Allen-Cahn partial differential equations via truncated full-history recursive multilevel Picard approximations
• Convergence rates for optimised adaptive importance samplers
• Non-convergence of stochastic gradient descent in the training of deep neural networks
• Solving the Kolmogorov PDE by means of deep learning
• Proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• Adaptive importance sampling with forward-backward stochastic differential equations
• Overcoming the curse of dimensionality in the approximative pricing of financial derivatives with default risks
• Controlled sequential Monte Carlo
• Variational Monte Carlo -- bridging concepts of machine learning and high-dimensional partial differential equations
• Deep relaxation: partial differential equations for optimizing deep neural networks
• Monte-Carlo methods and stochastic processes. From linear to non-linear
• Transformation of measure on Wiener space
• Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games. I: The ergodic case
• Solving high-dimensional optimal stopping problems using deep learning
• Overcoming the curse of dimensionality in the numerical approximation of semilinear parabolic partial differential equations
• Variational approach to rare event simulation using least-squares regression
• Data assimilation: the Schrödinger perspective
• Deep optimal stopping

Cited In (29)

• Numerical solutions of sea turtle population dynamics model by using restarting strategy of PINN-Adam
• Importance sampling for the empirical measure of weakly interacting diffusions
• Learning-based importance sampling via stochastic optimal control for stochastic reaction networks
• Neural parametric Fokker-Planck equation
• Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
• Actor-critic method for high dimensional static Hamilton-Jacobi-Bellman partial differential equations based on neural networks
• State-dependent Riccati equation feedback stabilization for nonlinear PDEs
• An overview on deep learning-based approximation methods for partial differential equations
• Connecting stochastic optimal control and reinforcement learning
• A neural network approach for stochastic optimal control
• Overcoming the timescale barrier in molecular dynamics: Transfer operators, variational principles and machine learning
• Approximation error analysis of some deep backward schemes for nonlinear PDEs
• Mobility Estimation for Langevin Dynamics Using Control Variates
• Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing
• Stein variational gradient descent: many-particle and long-time asymptotics
• Approximative policy iteration for exit time feedback control problems driven by stochastic differential equations using tensor train format
• Hamilton-Jacobi equations and mathematical morphology in pseudo-Riemannian manifolds
• Neural network architectures using min-plus algebra for solving certain high-dimensional optimal control problems and Hamilton-Jacobi PDEs
• Learning Koopman eigenfunctions of stochastic diffusions with optimal importance sampling and ISOKANN
• A Neural Network Approach to High-Dimensional Optimal Switching Problems with Jumps in Energy Markets
• Neural Control of Parametric Solutions for High-Dimensional Evolution PDEs
• Double-loop importance sampling for McKean-Vlasov stochastic differential equation
• Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning
• Approximating optimal feedback controllers of finite horizon control problems using hierarchical tensor formats
• Numerical methods for backward stochastic differential equations: a survey
• Approximation of optimal feedback controls for stochastic reaction-diffusion equations
• Bayesian learning via neural Schrödinger-Föllmer flows
• Error bounds for model reduction of feedback-controlled linear stochastic dynamics on Hilbert spaces
• Extensions of the deep Galerkin method

This page was built for publication: Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q825596)