The Principles of Deep Learning Theory
From MaRDI portal
Abstract: This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.
Recommendations
- Principles of artificial neural networks. Basic designs to deep learning
- scientific article; zbMATH DE number 4087481
- Neural Network Learning
- Computational methods for deep learning. Theoretic, practice and applications
- Deep Neural Network Approximation Theory
- Theoretical issues in deep networks
- scientific article; zbMATH DE number 926775
- Deep vs. shallow networks: an approximation theory perspective
- Computational Methods for Deep Learning
- ReLU deep neural networks from the hierarchical basis perspective
Cited in
(25)- Introduction to deep learning. From logical calculus to artificial intelligence
- \(p\)-adic statistical field theory and deep belief networks
- Unified field theoretical approach to deep and recurrent neuronal networks
- Asymptotics of representation learning in finite Bayesian neural networks*
- Geometry of deep learning. A signal processing perspective
- Deep learning architectures. A mathematical approach
- Deep learning in science
- Foundations of deep learning
- Inferring parameters of pyramidal neuron excitability in mouse models of Alzheimer's disease using biophysical modeling and deep learning
- Random neural networks in the infinite width limit as Gaussian processes
- On random matrices arising in deep neural networks: General I.I.D. case
- -Stable convergence of heavy-/light-tailed infinitely wide neural networks
- Principles for initialization and architecture selection in graph neural networks with ReLU activations
- Exploring fully-heavy tetraquarks through the CGAN framework: Mass and width
- Quantitative CLTs in deep neural networks
- Dynamic mean-field theory for continuous random networks
- Solution of the multimode nonlinear Schrödinger equation using physics-informed neural networks
- Dynamics of finite width kernel and prediction fluctuations in mean field neural networks
- Les houches lectures on deep learning at large and infinite width
- The geometry of the deep linear network
- The neural networks with tensor weights and emergent fermionic Wick rules in the large-width limit
- Catapult dynamics and phase transitions in quadratic nets
- Large and moderate deviations for Gaussian neural networks
- Scaling and renormalization in high-dimensional regression
- Deep neural networks: a formulation via non-Archimedean analysis
This page was built for publication: The Principles of Deep Learning Theory
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5070199)
