Smaller generalization error derived for a deep residual neural network compared with shallow networks

DOI10.1093/IMANUM/DRAC049MaRDI QIDQ6190811zbMATH OpenOpenAlexFDO

Authors Aku Kammonen, Jonas Kiessling, Petr Plecháč, Mattias Sandberg, Anders Szepessy, R. Tempone

Publication date 6 February 2024

Published in IMA Journal of Numerical Analysis (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/2010.01887

zbMATH Keywords

supervised learningerror estimatesresidual networkdeep random feature networkslayer-by-layer algorithm

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Randomized algorithms (68W20)

Abstract: Estimates of the generalization error are proved for a residual neural network with $L$ random Fourier features layers $$. An optimal distribution for the frequencies $(omeg{a}_{ellk},omega{\text{'}}_{ellk})$ of the random Fourier features $$ and $e^{mathrmiomega{\text{'}}_{ellk}cdotx}$ is derived. This derivation is based on the corresponding generalization error for the approximation of the function values $f(x)$. The generalization error turns out to be smaller than the estimate $|hatf{|}_{L^1(mathbb{R}^d)}^2/(KL)$ of the generalization error for random Fourier features with one hidden layer and the same total number of nodes $KL$, in the case the $L^{i}nfty$-norm of $f$ is much less than the $L^1$-norm of its Fourier transform $hatf$. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network. Promising performance of the proposed new algorithm is demonstrated in computational experiments.

Recommendations

• Rademacher complexity and the generalization error of residual networks
• Deep limits of residual neural networks
• An analysis of training and generalization errors in shallow and deep networks
• The generalization error of the minimum-norm solutions for over-parameterized neural networks
• Convergence analysis of deep residual networks

This page was built for publication: Smaller generalization error derived for a deep residual neural network compared with shallow networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6190811)