Fast generalization error bound of deep learning without scale invariance of activation functions

DOI10.1016/J.NEUNET.2020.05.033MaRDI QIDQ2055056zbMATH OpenOpenAlexDBLPWikidataFDO

Authors Yoshikazu Terada, Ryoma Hirose

Publication date 3 December 2021

Published in Neural Networks (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1907.10900

zbMATH Keywords

deep learningempirical risk minimizerexponential linear unitfast learning ratesigmoid activation function

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07)

Abstract: In theoretical analysis of deep learning, discovering which features of deep learning lead to good performance is an important task. In this paper, using the framework for analyzing the generalization error developed in Suzuki (2018), we derive a fast learning rate for deep neural networks with more general activation functions. In Suzuki (2018), assuming the scale invariance of activation functions, the tight generalization error bound of deep learning was derived. They mention that the scale invariance of the activation function is essential to derive tight error bounds. Whereas the rectified linear unit (ReLU; Nair and Hinton, 2010) satisfies the scale invariance, the other famous activation functions including the sigmoid and the hyperbolic tangent functions, and the exponential linear unit (ELU; Clevert et al., 2016) does not satisfy this condition. The existing analysis indicates a possibility that a deep learning with the non scale invariant activations may have a slower convergence rate of $O(1/sqrtn)$ when one with the scale invariant activations can reach a rate faster than $O(1/sqrtn)$. In this paper, without the scale invariance of activation functions, we derive the tight generalization error bound which is essentially the same as that of Suzuki (2018). From this result, at least in the framework of Suzuki (2018), it is shown that the scale invariance of the activation functions is not essential to get the fast rate of convergence. Simultaneously, it is also shown that the theoretical framework proposed by Suzuki (2018) can be widely applied for analysis of deep learning with general activation functions.

Recommendations

• Error bounds for approximations with deep ReLU networks
• On sharpness of an error bound for deep ReLU network approximation
• Error bounds for approximations with deep ReLU neural networks in \(W^{s , p}\) norms
• Approximation rates for neural networks with general activation functions
• Error bounds for ReLU networks with depth and width parameters
• Error estimates for DeepONets: a deep learning framework in infinite dimensions
• Adaptive approximation and generalization of deep neural network with intrinsic dimensionality
• A tight upper bound on the generalization error of feedforward neural networks
• On Lipschitz Bounds of General Convolutional Neural Networks
• High-dimensional dynamics of generalization error in neural networks

Cites work

• Approximation by superpositions of a sigmoidal function
• Breaking the curse of dimensionality with convex neural networks
• Empirical margin distributions and bounding the generalization error of combined classifiers
• Error bounds for approximations with deep ReLU networks
• Harmonic analysis of neural networks
• High-dimensional probability. An introduction with applications in data science
• scientific article; zbMATH DE number 49190 (Why is no real title available?)
• scientific article; zbMATH DE number 893887 (Why is no real title available?)
• scientific article; zbMATH DE number 7064043 (Why is no real title available?)
• scientific article; zbMATH DE number 3187905 (Why is no real title available?)
• Improving the sample complexity using global data
• Local Rademacher complexities
• Local Rademacher complexities and oracle inequalities in risk minimization. (2004 IMS Medallion Lecture). (With discussions and rejoinder)
• Multilayer feedforward networks are universal approximators
• Neural network with unbounded activation functions is universal approximator
• Nonparametric regression using deep neural networks with ReLU activation function
• On consistency of kernel density estimators for randomly censored data: Rates holding uniformly over adaptive intervals
• On the convergence rate of \(l_{p}\)-norm multiple kernel learning
• On the equivalence between kernel quadrature rules and random feature expansions
• On the expressive power of deep architectures
• Optimal approximation of piecewise smooth functions using deep ReLU neural networks
• Sharper bounds for Gaussian and empirical processes
• Statistical performance of support vector machines
• Support Vector Machines
• Theory of Reproducing Kernels
• Weak convergence and empirical processes. With applications to statistics

Cited in

(3)

• Over-parametrized deep neural networks minimizing the empirical risk do not generalize well
• An error analysis for deep binary classification with sigmoid loss
• Convergence analysis for over-parameterized deep learning

This page was built for publication: Fast generalization error bound of deep learning without scale invariance of activation functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2055056)