Full error analysis for the training of deep neural networks

DOI10.1142/S021902572150020XzbMATH Open1492.65132arXiv1910.00121OpenAlexW2977465327MaRDI QIDQ5083408FDOQ5083408

Christian Beck, Arnulf Jentzen, Benno Kuckuck

Publication date: 20 June 2022

Published in: Infinite Dimensional Analysis, Quantum Probability and Related Topics (Search for Journal in Brave)

Abstract: Deep learning algorithms have been applied very successfully in recent years to a range of problems out of reach for classical solution paradigms. Nevertheless, there is no completely rigorous mathematical error and convergence analysis which explains the success of deep learning algorithms. The error of a deep learning algorithm can in many situations be decomposed into three parts, the approximation error, the generalization error, and the optimization error. In this work we estimate for a certain deep learning algorithm each of these three errors and combine these three error estimates to obtain an overall error analysis for the deep learning algorithm under consideration. In particular, we thereby establish convergence with a suitable convergence speed for the overall error of the deep learning algorithm under consideration. Our convergence speed analysis is far from optimal and the convergence speed that we establish is rather slow, increases exponentially in the dimensions, and, in particular, suffers from the curse of dimensionality. The main contribution of this work is, instead, to provide a full error analysis (i) which covers each of the three different sources of errors usually emerging in deep learning algorithms and (ii) which merges these three sources of errors into one overall error estimate for the considered deep learning algorithm.

Full work available at URL: https://arxiv.org/abs/1910.00121

Recommendations

• An analysis of training and generalization errors in shallow and deep networks
• The Modern Mathematics of Deep Learning
• Mean Field Analysis of Deep Neural Networks
• Theoretical issues in deep networks
• scientific article; zbMATH DE number 6178473

zbMATH Keywords

artificial neural networksmachine learningconvergence analysisgeneralization errorapproximation erroroptimization errorfull error analysis

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Error analysis and interval analysis (65G99)

Cites Work

• Title not available (Why is that?)
• Universal approximation bounds for superpositions of a sigmoidal function
• Probability Inequalities for Sums of Bounded Random Variables
• On the mathematical foundations of learning
• Title not available (Why is that?)
• Understanding Machine Learning
• A distribution-free theory of nonparametric regression
• Local Rademacher complexities
• Title not available (Why is that?)
• Sets of finite perimeter and geometric variational problems. An introduction to geometric measure theory
• Multilayer feedforward networks are universal approximators
• Approximation by superpositions of a sigmoidal function
• Tractability of multivariate problems. Volume I: Linear information
• Tractability of multivariate problems. Volume II: Standard information for functionals.
• Title not available (Why is that?)
• Error bounds for approximation with neural networks
• Degree of approximation by neural and translation networks with a single hidden layer
• Approximation and estimation bounds for artificial neural networks
• Title not available (Why is that?)
• Approximation of functions and their derivatives: A neural network implementation with applications
• Title not available (Why is that?)
• On the approximation by single hidden layer feedforward neural networks with fixed weights
• Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Error bounds for approximations with deep ReLU networks
• A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
• Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates
• Neural Networks for Localized Approximation
• Deep learning in high dimension: Neural network expression rates for generalized polynomial chaos expansions in UQ
• Optimal Approximation with Sparsely Connected Deep Neural Networks
• Solving the Kolmogorov PDE by means of deep learning
• Proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
• Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black--Scholes Partial Differential Equations
• Deep Neural Networks Algorithms for Stochastic Control Problems on Finite Horizon: Convergence Analysis
• A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Strong error analysis for stochastic gradient descent optimization algorithms
• Deep vs. shallow networks: an approximation theory perspective
• Approximation spaces of deep neural networks
• General multilevel adaptations for stochastic approximation algorithms of Robbins-Monro and Polyak-Ruppert type
• Provable approximation properties for deep neural networks
• Exponential convergence of the deep neural network approximation for analytic functions
• Error bounds for approximations with deep ReLU neural networks in Ws,p norms
• Topological properties of the set of functions generated by neural networks of fixed size
• Gradient descent optimizes over-parameterized deep ReLU networks
• Nonlinear approximation via compositions
• Rectified deep neural networks overcome the curse of dimensionality for nonsmooth value functions in zero-sum games of nonlinear stiff systems
• Equivalence of approximation by convolutional neural networks and fully-connected networks
• Title not available (Why is that?)
• Deep Neural Network Approximation Theory
• A theoretical analysis of deep neural networks and parametric PDEs
• Deep Network Approximation Characterized by Number of Neurons
• On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex Case
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• DNN expression rate analysis of high-dimensional PDEs: application to option pricing
• Universal approximations of invariant maps by neural networks

Cited In (12)

• Deep learning based on randomized quasi-Monte Carlo method for solving linear Kolmogorov partial differential equation
• An analysis of training and generalization errors in shallow and deep networks
• Error analysis for deep neural network approximations of parametric hyperbolic conservation laws
• Numerical analysis of physics-informed neural networks and related models in physics-informed machine learning
• Learning the random variables in Monte Carlo simulations with stochastic gradient descent: Machine learning for parametric PDEs and financial derivative pricing
• Strong overall error analysis for the training of artificial neural networks via random initializations
• Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation
• Lower bounds for artificial neural network approximations: a proof that shallow neural networks fail to overcome the curse of dimensionality
• Error analysis for empirical risk minimization over clipped ReLU networks in solving linear Kolmogorov partial differential equations
• Error Analysis and Improving the Accuracy of Winograd Convolution for Deep Neural Networks
• A proof of convergence for gradient descent in the training of artificial neural networks for constant target functions
• A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

This page was built for publication: Full error analysis for the training of deep neural networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5083408)