Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black--Scholes Partial Differential Equations

DOI10.1137/19M125649XzbMath1480.60191arXiv1809.03062MaRDI QIDQ5037569

Arnulf Jentzen, Julius Berner, Philipp Grohs

Publication date: 1 March 2022

Published in: SIAM Journal on Mathematics of Data Science (Search for Journal in Brave)

Abstract: The development of new classification and regression algorithms based on empirical risk minimization (ERM) over deep neural network hypothesis classes, coined deep learning, revolutionized the area of artificial intelligence, machine learning, and data analysis. In particular, these methods have been applied to the numerical solution of high-dimensional partial differential equations with great success. Recent simulations indicate that deep learning-based algorithms are capable of overcoming the curse of dimensionality for the numerical solution of Kolmogorov equations, which are widely used in models from engineering, finance, and the natural sciences. The present paper considers under which conditions ERM over a deep neural network hypothesis class approximates the solution of a $d$-dimensional Kolmogorov equation with affine drift and diffusion coefficients and typical initial values arising from problems in computational finance up to error $varepsilon$. We establish that, with high probability over draws of training samples, such an approximation can be achieved with both the size of the hypothesis class and the number of training samples scaling only polynomially in $d$ and $varepsilon^{-1}$. It can be concluded that ERM over deep neural network hypothesis classes overcomes the curse of dimensionality for the numerical solution of linear Kolmogorov equations with affine coefficients.

Full work available at URL: https://arxiv.org/abs/1809.03062

zbMATH Keywords

curse of dimensionalityKolmogorov equationgeneralization errorempirical risk minimizationdeep learning

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Applications of stochastic analysis (to PDEs, etc.) (60H30) Numerical solutions to stochastic differential and integral equations (65C30) Neural nets and related approaches to inference from stochastic processes (62M45)

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Oracle inequalities in empirical risk minimization and sparse recovery problems. École d'Été de Probabilités de Saint-Flour XXXVIII-2008.
• Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
• Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII -- 2003.
• Provable approximation properties for deep neural networks
• The Deep Ritz Method: a deep learning-based numerical algorithm for solving variational problems
• A distribution-free theory of nonparametric regression
• DGM: a deep learning algorithm for solving partial differential equations
• Topological properties of the set of functions generated by neural networks of fixed size
• Solving the Kolmogorov PDE by means of deep learning
• Proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
• Error bounds for approximations with deep ReLU networks
• Asymptotic expansion as prior knowledge in deep learning method for high dimensional BSDEs
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Loss of regularity for Kolmogorov equations
• Stochastic simulation and Monte Carlo methods. Mathematical foundations of stochastic simulation
• Local Rademacher complexities
• On the mathematical foundations of learning
• Learning Theory
• Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ
• Neural Network Learning
• A mean field view of the landscape of two-layer neural networks
• Solving high-dimensional partial differential equations using deep learning
• Deep Neural Network Approximation Theory
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
• Deep optimal stopping
• Tools for computational finance.
• Error bounds for approximation with neural networks

Related Items (65)

Convergence bounds for empirical nonlinear least-squares ⋮ Non intrusive reduced order modeling of parametrized PDEs by kernel POD and neural networks ⋮ Convergence of a Robust Deep FBSDE Method for Stochastic Control ⋮ Convergence of Physics-Informed Neural Networks Applied to Linear Second-Order Elliptic Interface Problems ⋮ A new efficient approximation scheme for solving high-dimensional semilinear PDEs: control variate method for deep BSDE solver ⋮ Metric entropy limits on recurrent neural network learning of linear dynamical systems ⋮ Error estimates for deep learning methods in fluid dynamics ⋮ Deep neural network approximations for solutions of PDEs based on Monte Carlo algorithms ⋮ Quantitative Approximation Results for Complex-Valued Neural Networks ⋮ Overcoming the curse of dimensionality in the numerical approximation of parabolic partial differential equations with gradient-dependent nonlinearities ⋮ Approximations with deep neural networks in Sobolev time-space ⋮ Imaging conductivity from current density magnitude using neural networks* ⋮ Full error analysis for the training of deep neural networks ⋮ Unbiased Deep Solvers for Linear Parametric PDEs ⋮ Towards fast weak adversarial training to solve high dimensional parabolic partial differential equations using XNODE-WAN ⋮ Physics and equality constrained artificial neural networks: application to forward and inverse problems with multi-fidelity data fusion ⋮ Deep Neural Network Surrogates for Nonsmooth Quantities of Interest in Shape Uncertainty Quantification ⋮ Simultaneous neural network approximation for smooth functions ⋮ Three ways to solve partial differential equations with neural networks — A review ⋮ Monte Carlo simulation of SDEs using GANs ⋮ Sparse polynomial approximations for affine parametric saddle point problems ⋮ Physics-informed neural networks for approximating dynamic (hyperbolic) PDEs of second order in time: error analysis and algorithms ⋮ Solving non-linear Kolmogorov equations in large dimensions by using deep learning: a numerical comparison of discretization schemes ⋮ Exponential ReLU neural network approximation rates for point and edge singularities ⋮ Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation ⋮ Active learning based sampling for high-dimensional nonlinear partial differential equations ⋮ Space-time error estimates for deep neural network approximations for differential equations ⋮ An extreme learning machine-based method for computational PDEs in higher dimensions ⋮ Lower bounds for artificial neural network approximations: a proof that shallow neural networks fail to overcome the curse of dimensionality ⋮ Numerical methods for backward stochastic differential equations: a survey ⋮ Neural network approximation and estimation of classifiers with classification boundary in a Barron class ⋮ Deep Weak Approximation of SDEs: A Spatial Approximation Scheme for Solving Kolmogorov Equations ⋮ Solving Kolmogorov PDEs without the curse of dimensionality via deep learning and asymptotic expansion with Malliavin calculus ⋮ Overcoming the curse of dimensionality in the approximative pricing of financial derivatives with default risks ⋮ Adaptive neural network surrogate model for solving the implied volatility of time-dependent American option via Bayesian inference ⋮ An overview on deep learning-based approximation methods for partial differential equations ⋮ Greedy training algorithms for neural networks and applications to PDEs ⋮ Non-Linear Interactions and Exchange Rate Prediction: Empirical Evidence Using Support Vector Regression ⋮ Strong overall error analysis for the training of artificial neural networks via random initializations ⋮ Deep neural network expressivity for optimal stopping problems ⋮ Overcoming the curse of dimensionality in the numerical approximation of high-dimensional semilinear elliptic partial differential equations ⋮ Error analysis for deep neural network approximations of parametric hyperbolic conservation laws ⋮ Deep ReLU network expression rates for option prices in high-dimensional, exponential Lévy models ⋮ Solving PDEs on unknown manifolds with machine learning ⋮ Numerical analysis of physics-informed neural networks and related models in physics-informed machine learning ⋮ Deep learning based on randomized quasi-Monte Carlo method for solving linear Kolmogorov partial differential equation ⋮ Deep Splitting Method for Parabolic PDEs ⋮ Proof of the theory-to-practice gap in deep learning via sampling complexity bounds for neural network approximation spaces ⋮ Generalization of PINNs for elliptic interface problems ⋮ Computing Lyapunov functions using deep neural networks ⋮ Parameter identification by deep learning of a material model for granular media ⋮ Sampling complexity of deep approximation spaces ⋮ Error analysis for empirical risk minimization over clipped ReLU networks in solving linear Kolmogorov partial differential equations ⋮ Solving the Kolmogorov PDE by means of deep learning ⋮ Variational Monte Carlo -- bridging concepts of machine learning and high-dimensional partial differential equations ⋮ Optimally weighted loss functions for solving PDEs with neural networks ⋮ Enhancing the momentum strategy through deep regression ⋮ RICAM, the Johann Radon Institute for Computational and Applied Mathematics ⋮ Asymptotic expansion as prior knowledge in deep learning method for high dimensional BSDEs ⋮ Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs ⋮ Stable recovery of entangled weights: towards robust identification of deep neural networks from minimal samples ⋮ A measure theoretical approach to the mean-field maximum principle for training NeurODEs ⋮ Algorithms for solving high dimensional PDEs: from nonlinear Monte Carlo to machine learning ⋮ DNN expression rate analysis of high-dimensional PDEs: application to option pricing ⋮ A theoretical analysis of deep neural networks and parametric PDEs

Uses Software

• MNIST
• Adam
• DGM

This page was built for publication: Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black--Scholes Partial Differential Equations