Convergence of deep convolutional neural networks
From MaRDI portal
Abstract: Convergence of deep neural networks as the depth of the networks tends to infinity is fundamental in building the mathematical foundation for deep learning. In a previous study, we investigated this question for deep ReLU networks with a fixed width. This does not cover the important convolutional neural networks where the widths are increasing from layer to layer. For this reason, we first study convergence of general ReLU networks with increasing widths and then apply the results obtained to deep convolutional neural networks. It turns out the convergence reduces to convergence of infinite products of matrices with increasing sizes, which has not been considered in the literature. We establish sufficient conditions for convergence of such infinite products of matrices. Based on the conditions, we present sufficient conditions for piecewise convergence of general deep ReLU networks with increasing widths, and as well as pointwise convergence of deep ReLU convolutional neural networks.
Recommendations
- Convergence analysis of deep residual networks
- Convergence rates of deep ReLU networks for multiclass classification
- Universality of deep convolutional neural networks
- Approximation properties of deep ReLU CNNs
- Gradient descent on infinitely wide neural networks: global convergence and generalization
Cites work
- Deep learning
- Deep network approximation characterized by number of neurons
- Deep network with approximation error being reciprocal of width to power of square root of depth
- Deep Neural Network Approximation Theory
- Equivalence of approximation by convolutional neural networks and fully-connected networks
- Error bounds for approximations with deep ReLU networks
- Error bounds for deep ReLU networks using the Kolmogorov-Arnold superposition theorem
- Exponential convergence of the deep neural network approximation for analytic functions
- scientific article; zbMATH DE number 1324223 (Why is no real title available?)
- scientific article; zbMATH DE number 1745905 (Why is no real title available?)
- scientific article; zbMATH DE number 1881986 (Why is no real title available?)
- scientific article; zbMATH DE number 1889798 (Why is no real title available?)
- scientific article; zbMATH DE number 3196283 (Why is no real title available?)
- Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators
- Nonlinear approximation and (deep) ReLU networks
- On the convergence of infinite products of matrices
- Parseval proximal neural networks
- The gap between theory and practice in function approximation with deep neural networks
- Universality of deep convolutional neural networks
Cited in
(12)- Theory of deep convolutional neural networks. II: Spherical analysis
- MG-CNN: a deep CNN to predict saddle points of matrix games
- Globally Convergent Multilevel Training of Deep Residual Networks
- Convergence analysis of deep residual networks
- Global convergence in learning fully-connected ReLU networks via un-rectifying based on the augmented Lagrangian approach
- Deep neural network solutions for oscillatory Fredholm integral equations
- Function space and critical points of linear convolutional networks
- On the density of translation networks defined on the unit ball
- Adaptive multi-grade deep learning for highly oscillatory Fredholm integral equations of the second kind
- Successive affine learning for deep neural networks
- Lumped disturbances estimation-based inverse dynamic cooperation control for uncertain arm manipulators
- Applications of Bregman distance function to improve the performance of the plant leaf image retrieval system
This page was built for publication: Convergence of deep convolutional neural networks
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6077046)
