Smaller generalization error derived for a deep residual neural network compared with shallow networks

DOI10.1093/IMANUM/DRAC049arXiv2010.01887OpenAlexW3153985548MaRDI QIDQ6190811FDOQ6190811

Mattias Sandberg, Anders Szepessy, Jonas Kiessling, Aku Kammonen, R. Tempone, Petr Plecháč

Publication date: 6 February 2024

Published in: IMA Journal of Numerical Analysis (Search for Journal in Brave)

Abstract: Estimates of the generalization error are proved for a residual neural network with $L$ random Fourier features layers $$. An optimal distribution for the frequencies $(omeg{a}_{ellk},omega{\text{'}}_{ellk})$ of the random Fourier features $$ and $e^{mathrmiomega{\text{'}}_{ellk}cdotx}$ is derived. This derivation is based on the corresponding generalization error for the approximation of the function values $f(x)$. The generalization error turns out to be smaller than the estimate $|hatf{|}_{L^1(mathbb{R}^d)}^2/(KL)$ of the generalization error for random Fourier features with one hidden layer and the same total number of nodes $KL$, in the case the $L^{i}nfty$-norm of $f$ is much less than the $L^1$-norm of its Fourier transform $hatf$. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network. Promising performance of the proposed new algorithm is demonstrated in computational experiments.

Full work available at URL: https://arxiv.org/abs/2010.01887

zbMATH Keywords

supervised learningerror estimatesresidual networkdeep random feature networkslayer-by-layer algorithm

Mathematics Subject Classification ID

Artificial neural networks and deep learning (68T07) Randomized algorithms (68W20)

This page was built for publication: Smaller generalization error derived for a deep residual neural network compared with shallow networks

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6190811)