Actor-critic method for high dimensional static Hamilton-Jacobi-Bellman partial differential equations based on neural networks
From MaRDI portal
Publication:5021407
actor-critic methodsHamilton-Jacobi-Bellman equationshigh dimensional partial differential equationsstochastic control
Nonlinear elliptic equations (35J60) Hamilton-Jacobi equations in optimal control and differential games (49L12) Numerical solutions to stochastic differential and integral equations (65C30) Numerical mathematical programming methods (65K05) Probabilistic methods, particle methods, etc. for initial value and initial-boundary value problems involving PDEs (65M75) Artificial neural networks and deep learning (68T07) Optimal stochastic control (93E20)
Abstract: We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive step size scheme to improve the accuracy near the domain boundary. Numerical examples up to spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, the diffusive Eikonal equations, and fully nonlinear elliptic PDEs derived from a regulator problem are presented to validate the effectiveness of our proposed method.
Recommendations
- Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space
- Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
- Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
- Tensor decomposition methods for high-dimensional Hamilton-Jacobi-Bellman equations
- Overcoming the curse of dimensionality for some Hamilton-Jacobi partial differential equations via neural network architectures
Cites work
- A derivative-free method for solving elliptic partial differential equations with deep neural networks
- A time-dependent Hamilton-Jacobi formulation of reachable sets for continuous dynamic games
- Adapted solution of a backward stochastic differential equation
- Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
- Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning
- Approximate solutions to the time-invariant Hamilton-Jacobi-Bellman equation
- Convergence of the deep BSDE method for coupled FBSDEs
- Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
- Deep optimal stopping
- Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations
- Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation
- HJB-POD-Based Feedback Design for the Optimal Control of Evolution Problems
- scientific article; zbMATH DE number 140601 (Why is no real title available?)
- scientific article; zbMATH DE number 1325009 (Why is no real title available?)
- scientific article; zbMATH DE number 1121855 (Why is no real title available?)
- scientific article; zbMATH DE number 2199827 (Why is no real title available?)
- Machine learning for semi linear PDEs
- Mitigating the curse of dimensionality: sparse grid characteristics method for optimal feedback control and HJB equations
- Natural actor-critic algorithms
- Neural networks-based backward scheme for fully nonlinear PDEs
- Numerical solution of Hamilton-Jacobi-Bellman equations by an upwind finite volume method
- Numerical solution of Hamilton–Jacobi–Bellman equations by an exponentially fitted finite volume method
- On reachability and minimum cost optimal control
- On the convergence rate of approximation schemes for Hamilton-Jacobi-Bellman Equations
- OnActor-Critic Algorithms
- Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem
- Overapproximating reachable sets by Hamilton-Jacobi projections
- Polynomial approximation of high-dimensional Hamilton-Jacobi-Bellman equations and applications to feedback control of semilinear parabolic PDEs
- Probabilistic interpretation for systems of quasilinear parabolic partial differential equations
- Recurrent neural networks for stochastic control problems with delay
- Reinforcement learning. An introduction
- Solving Dirichlet problems numerically using the Feynman-Kac representation
- Solving high-dimensional eigenvalue problems using deep neural networks: a diffusion Monte Carlo like approach
- Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space
- Solving high-dimensional partial differential equations using deep learning
- Tensor decomposition methods for high-dimensional Hamilton-Jacobi-Bellman equations
- User’s guide to viscosity solutions of second order partial differential equations
- Weak approximation of killed diffusion using Euler schemes.
Cited in
(43)- On some neural network architectures that can represent viscosity solutions of certain high dimensional Hamilton-Jacobi partial differential equations
- Extensions of the deep Galerkin method
- Deep reinforcement learning of viscous incompressible flow
- Solving non-linear Kolmogorov equations in large dimensions by using deep learning: a numerical comparison of discretization schemes
- Adaptive deep learning for high-dimensional Hamilton-Jacobi-Bellman equations
- Tensor decomposition methods for high-dimensional Hamilton-Jacobi-Bellman equations
- SympOCnet: Solving Optimal Control Problems with Applications to High-Dimensional Multiagent Path Planning Problems
- Approximating optimal feedback controllers of finite horizon control problems using hierarchical tensor formats
- Approximative policy iteration for exit time feedback control problems driven by stochastic differential equations using tensor train format
- Data-Driven Tensor Train Gradient Cross Approximation for Hamilton–Jacobi–Bellman Equations
- Deep reinforcement learning in finite-horizon to explore the most probable transition pathway
- Dynamic order allocation in a duopoly hybrid workforce of competition: a machine learning approach
- Lax-Oleinik-type formulas and efficient algorithms for certain high-dimensional optimal control problems
- Connecting stochastic optimal control and reinforcement learning
- Recent developments in machine learning methods for stochastic control and games
- A multilinear HJB-POD method for the optimal control of PDEs on a tree structure
- Solution theory of Hamilton-Jacobi-Bellman equations in spectral Barron spaces
- MscaleFNO: multi-scale Fourier neural operator learning for oscillatory functions and wave scattering problems
- Reverse diffusion sampling with tensor train approximations of Hamilton-Jacobi-Bellman equations
- Deep Picard iteration for high-dimensional nonlinear PDEs
- Neural implicit solution formula for efficiently solving Hamilton-Jacobi equations
- Score-based neural ordinary differential equations for computing mean field control problems
- Data-driven control for stochastic linear-quadratic optimal problem with completely unknown dynamics
- Space-time deep neural network approximations for high-dimensional partial differential equations
- Multiagent relative investment games in a jump diffusion market with deep reinforcement learning algorithm
- A policy gradient framework for stochastic optimal control problems with global convergence guarantee
- An actor critic machine learning method for free terminal time optimal control
- Interpolating between BSDEs and PINNs: deep learning for elliptic and parabolic boundary value problems
- SOC-MartNet: a martingale neural network for the Hamilton-Jacobi-Bellman equation without explicit \(\inf_{u\in U}H\) in stochastic optimal controls
- A time-dependent symplectic network for nonconvex path planning problems with linear and nonlinear dynamics
- A deep learning algorithm for computing mean field control problems via forward-backward score dynamics
- A comparison study of supervised learning techniques for the approximation of high dimensional functions and feedback control
- State dependent Riccati for dynamic boundary control to optimize irrigation in Richards' equation framework
- Neural network approaches for parameterized optimal control
- The state-dependent Riccati equation in nonlinear optimal control: analysis and numerical approximation
- Deep random difference method for high-dimensional quasilinear parabolic partial differential equations
- On Bellman equations for continuous-time policy evaluation: high-order discretization and function approximation
- A new numerical scheme for stochastic control problems of jump-diffusion equations by means of Runge-Kutta method
- Dynamical low-rank approximation strategies for nonlinear feedback control problems
- Beyond Phase Reduction: Amplitude Collapse in Optimal Control of Coupled Oscillators
- Simulating Fokker-Planck equations via mean field control of score-based normalizing flows
- Overcoming the curse of dimensionality for some Hamilton-Jacobi partial differential equations via neural network architectures
- Solving high-dimensional Hamilton-Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space
This page was built for publication: Actor-critic method for high dimensional static Hamilton-Jacobi-Bellman partial differential equations based on neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5021407)
