Scaling description of generalization with number of parameters in deep learning

DOI10.1088/1742-5468/AB633CMaRDI QIDQ5856249zbMATH OpenOpenAlexFDO

Authors Mario Geiger, A. Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane D'Ascoli, Clément Hongler, Matthieu Wyart, Giulio Biroli

Publication date 25 March 2021

Published in Journal of Statistical Mechanics: Theory and Experiment (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1901.01608

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Neural nets applied to problems in time-dependent statistical mechanics (82C32)

Abstract: Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{\ast }$. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $$ of the neural net output function $f_N$ around its expectation $$. These affect the generalization error $epsilo{n}_N$ for classification: under natural assumptions, it decays to a plateau value $epsilo{n}_{infty}$ in a power-law fashion $sim{N}^{-1/2}$. This description breaks down at a so-called jamming transition $N=N^{\ast }$. At this threshold, we argue that $|f_N|$ diverges. This result leads to a plausible explanation for the cusp in test error known to occur at $N^{\ast }$. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond $N^{\ast }$, and averaging their outputs.

Recommendations

• A jamming transition from under- to over-parametrization affects generalization in deep learning
• High-dimensional dynamics of generalization error in neural networks
• An analysis of training and generalization errors in shallow and deep networks
• Over-parametrized deep neural networks minimizing the empirical risk do not generalize well
• Deep learning: a statistical viewpoint

Cites work

• Bayesian learning for neural networks
• Comparing dynamics: deep neural networks versus glassy systems
• Reconciling modern machine-learning practice and the classical bias-variance trade-off
• Statistical mechanics of learning
• The implicit bias of gradient descent on separable data
• The simplest model of jamming

Cited in

(29)

• Over-parametrized deep neural networks minimizing the empirical risk do not generalize well
• Normalization effects on shallow neural networks and related asymptotic expansions
• A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
• Surprises in high-dimensional ridgeless least squares interpolation
• Landscape and training regimes in deep learning
• Free dynamics of feature learning processes
• Geometric compression of invariant manifolds in neural networks
• Triple descent and the two kinds of overfitting: where and why do they appear?*
• An analytic theory of shallow networks dynamics for hinge loss classification*
• The inverse variance-flatness relation in stochastic gradient descent is critical for finding flat minima
• Overparameterization and generalization error: weighted trigonometric interpolation
• What causes the test error? Going beyond bias-variance via ANOVA
• A jamming transition from under- to over-parametrization affects generalization in deep learning
• Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks
• Large-dimensional random matrix theory and its applications in deep learning and wireless communications
• High-Dimensional Analysis of Double Descent for Linear Regression with Random Projections
• Harmonic analysis of network systems via kernels and their boundary realizations
• A Generalization Gap Estimation for Overparameterized Models via the Langevin Functional Variance
• Normalization effects on deep neural networks
• The common intuition to transfer learning can win or lose: case studies for linear regression
• Fluctuations, bias, variance and ensemble of learners: exact asymptotics for convex losses in high-dimension
• Learning sparse features can lead to overfitting in neural networks
• Redundant representations help generalization in wide neural networks
• Limit shapes and large deviations in classical and quantum neural networks
• The generalization error of max-margin linear classifiers: benign overfitting and high dimensional asymptotics in the overparametrized regime
• Universality laws for Gaussian mixtures in generalized linear models
• Dynamical loss functions shape landscape topography and improve learning in artificial neural networks
• Scaling and renormalization in high-dimensional regression
• The nuclear route: sharp asymptotics of ERM in overparameterized quadratic networks

Describes a project that uses

Uses Software

• ImageNet
• AlexNet
• MNIST
• Adam

This page was built for publication: Scaling description of generalization with number of parameters in deep learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5856249)