Normalization effects on shallow neural networks and related asymptotic expansions

DOI10.3934/FODS.2021013MaRDI QIDQ2072629zbMATH OpenOpenAlexFDO

Authors Jiahui Yu, Konstantinos Spiliopoulos

Publication date 26 January 2022

Published in Foundations of Data Science (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2011.10487

zbMATH Keywords

asymptotic expansionsinitializationmachine learningneural networksnormalization effectout-of-sample performance

Mathematics Subject Classification ID

Central limit and other weak theorems (60F05) Stochastic processes (60G99) Neural nets and related approaches to inference from stochastic processes (62M45) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: We consider shallow (single hidden layer) neural networks and characterize their performance when trained with stochastic gradient descent as the number of hidden units $N$ and gradient descent steps grow to infinity. In particular, we investigate the effect of different scaling schemes, which lead to different normalizations of the neural network, on the network's statistical output, closing the gap between the $1/sqrtN$ and the mean-field $1/N$ normalization. We develop an asymptotic expansion for the neural network's statistical output pointwise with respect to the scaling parameter as the number of hidden units grows to infinity. Based on this expansion, we demonstrate mathematically that to leading order in $N$, there is no bias-variance trade off, in that both bias and variance (both explicitly characterized) decrease as the number of hidden units increases and time grows. In addition, we show that to leading order in $N$, the variance of the neural network's statistical output decays as the implied normalization by the scaling parameter approaches the mean field normalization. Numerical studies on the MNIST and CIFAR10 datasets show that test and train accuracy monotonically improve as the neural network's normalization gets closer to the mean field normalization.

Recommendations

• Mean Field Analysis of Deep Neural Networks
• Theoretical issues in deep networks
• Mean field analysis of neural networks: a central limit theorem
• Disentangling feature and lazy training in deep neural networks
• Mean field analysis of neural networks: a law of large numbers

Cites work

• A mean field view of the landscape of two-layer neural networks
• Approximation and estimation bounds for artificial neural networks
• Deep learning
• DGM: a deep learning algorithm for solving partial differential equations
• Gradient descent optimizes over-parameterized deep ReLU networks
• scientific article; zbMATH DE number 3951715 (Why is no real title available?)
• scientific article; zbMATH DE number 1972910 (Why is no real title available?)
• Large deviations and mean-field theory for asymmetric random recurrent neural networks
• Machine learning strategies for systems with invariance properties
• Mean field analysis of neural networks: a central limit theorem
• Mean field analysis of neural networks: a law of large numbers
• Multilayer feedforward networks are universal approximators
• Nonlinearity creates linear independence
• Reynolds averaged turbulence modelling using deep neural networks with embedded invariance
• Scaling description of generalization with number of parameters in deep learning
• Universal features of price formation in financial markets: perspectives from deep learning

Cited in

(3)

• Normalization effects on deep neural networks
• Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent
• Kernel limit for a class of recurrent neural networks trained on ergodic data sequences

Describes a project that uses

Uses Software

• DeepFace
• CIFAR
• DGM

This page was built for publication: Normalization effects on shallow neural networks and related asymptotic expansions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2072629)