Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation
From MaRDI portal
Publication:6107984
Abstract: In spite of the accomplishments of deep learning based algorithms in numerous applications and very broad corresponding research interest, at the moment there is still no rigorous understanding of the reasons why such algorithms produce useful results in certain situations. A thorough mathematical analysis of deep learning based algorithms seems to be crucial in order to improve our understanding and to make their implementation more effective and efficient. In this article we provide a mathematically rigorous full error analysis of deep learning based empirical risk minimisation with quadratic loss function in the probabilistically strong sense, where the underlying deep neural networks are trained using stochastic gradient descent with random initialisation. The convergence speed we obtain is presumably far from optimal and suffers under the curse of dimensionality. To the best of our knowledge, we establish, however, the first full error analysis in the scientific literature for a deep learning based algorithm in the probabilistically strong sense and, moreover, the first full error analysis in the scientific literature for a deep learning based algorithm where stochastic gradient descent with random initialisation is the employed optimisation method.
Recommendations
- Strong overall error analysis for the training of artificial neural networks via random initializations
- Full error analysis for the training of deep neural networks
- Non-convergence of stochastic gradient descent in the training of deep neural networks
- Strong error analysis for stochastic gradient descent optimization algorithms
- Mean Field Analysis of Deep Neural Networks
Cites work
- A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
- A distribution-free theory of nonparametric regression
- A priori estimates of the population risk for two-layer neural networks
- A proof that rectified deep neural networks overcome the curse of dimensionality in the numerical approximation of semilinear heat equations
- A Stochastic Approximation Method
- Analysis of the generalization error: empirical risk minimization over deep artificial neural networks overcomes the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations
- Approximation and estimation bounds for artificial neural networks
- Approximation by superpositions of a sigmoidal function
- Approximation of functions and their derivatives: A neural network implementation with applications
- Bounds for the ratio of two gamma functions
- Breaking the curse of dimensionality with convex neural networks
- Convergence in Hölder norms with applications to Monte Carlo methods in infinite dimensions
- Convergence rates for the stochastic gradient descent method for non-convex objective functions
- Deep learning
- Deep learning in high dimension: neural network expression rates for generalized polynomial chaos expansions in UQ
- Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
- Deep neural network approximations for solutions of PDEs based on Monte Carlo algorithms
- Deep vs. shallow networks: an approximation theory perspective
- Degree of approximation by neural and translation networks with a single hidden layer
- DGM: a deep learning algorithm for solving partial differential equations
- DNN expression rate analysis of high-dimensional PDEs: application to option pricing
- Error bounds for approximation with neural networks
- Error bounds for approximations with deep ReLU networks
- Exponential convergence of the deep neural network approximation for analytic functions
- Full error analysis for the training of deep neural networks
- General multilevel adaptations for stochastic approximation algorithms of Robbins-Monro and Polyak-Ruppert type
- Gradient descent optimizes over-parameterized deep ReLU networks
- scientific article; zbMATH DE number 3126094 (Why is no real title available?)
- scientific article; zbMATH DE number 671791 (Why is no real title available?)
- scientific article; zbMATH DE number 1083116 (Why is no real title available?)
- scientific article; zbMATH DE number 1405266 (Why is no real title available?)
- scientific article; zbMATH DE number 1420699 (Why is no real title available?)
- Infinite dimensional analysis. A hitchhiker's guide.
- Local Rademacher complexities
- Lower error bounds for the stochastic gradient descent optimization algorithm: sharp convergence rates for slowly and fast decaying learning rates
- Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations
- Multilayer feedforward networks are universal approximators
- Neural Networks for Localized Approximation
- New error bounds for deep ReLU networks using sparse grids
- Nonlinear approximation via compositions
- On the approximation by single hidden layer feedforward neural networks with fixed weights
- On the mathematical foundations of learning
- Optimal approximation of piecewise smooth functions using deep ReLU neural networks
- Optimal approximation with sparsely connected deep neural networks
- Proof that deep artificial neural networks overcome the curse of dimensionality in the numerical approximation of Kolmogorov partial differential equations with constant diffusion and nonlinear drift coefficients
- Provable approximation properties for deep neural networks
- Some Elementary Inequalities Relating to the Gamma and Incomplete Gamma Function
- Space-time error estimates for deep neural network approximations for differential equations
- Strong error analysis for stochastic gradient descent optimization algorithms
- Tractability of multivariate problems. Volume I: Linear information
- Tractability of multivariate problems. Volume II: Standard information for functionals.
- Understanding machine learning. From theory to algorithms
- Uniform error estimates for artificial neural network approximations for heat equations
- Universal approximation bounds for superpositions of a sigmoidal function
Cited in
(9)- Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness
- An analysis of training and generalization errors in shallow and deep networks
- Error Analysis and Improving the Accuracy of Winograd Convolution for Deep Neural Networks
- Corrections to “Deep Neural Networks With Random Gaussian Weights: A Universal Classification Strategy?” [Jul 1, 2016 3444-3457]
- Strong overall error analysis for the training of artificial neural networks via random initializations
- Generalization error analysis of deep backward dynamic programming for solving nonlinear PDEs
- DRM revisited: a complete error analysis
- Physics-informed neural fractional differential equations
- Convergence of stochastic gradient descent schemes for Łojasiewicz-landscapes
This page was built for publication: Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6107984)
