Comparing dynamics: deep neural networks versus glassy systems
From MaRDI portal
Neural nets applied to problems in time-dependent statistical mechanics (82C32) Dynamics of disordered systems (random Ising systems, etc.) in time-dependent statistical mechanics (82C44) Statistical mechanics of random media, disordered materials (including liquid crystals and spin glasses) (82D30)
Abstract: We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.
Recommendations
- Archetypal landscapes for deep neural networks
- Disentangling feature and lazy training in deep neural networks
- High-dimensional dynamics of generalization error in neural networks
- Shaping the learning landscape in neural networks around wide flat minima
- A jamming transition from under- to over-parametrization affects generalization in deep learning
Cites work
- Activated aging dynamics and effective trap model description in the random energy model
- Cugliandolo-Kurchan equations for dynamics of spin-glasses
- Determining computational complexity from characteristic ``phase transitions
- Gibbs states and the set of solutions of random constraint satisfaction problems
- scientific article; zbMATH DE number 3665495 (Why is no real title available?)
- Phase space geometry and slow dynamics
- Rigorous inequalities between length and time scales in glassy systems
- Spectral gap estimates in mean field spin glasses
- Spin-glass theory for pedestrians
Cited in
(19)- An emergent autonomous flow for mean-field spin glasses
- A spin glass model for the loss surfaces of generative adversarial networks
- Landscape and training regimes in deep learning
- The loss surfaces of neural networks with general activation functions
- Maximal relevance and optimal learning machines
- An analytic theory of shallow networks dynamics for hinge loss classification*
- scientific article; zbMATH DE number 7626737 (Why is no real title available?)
- Gradient descent dynamics and the jamming transition in infinite dimensions
- Archetypal landscapes for deep neural networks
- Ensemble inference methods for models with noisy and expensive likelihoods
- Scaling description of generalization with number of parameters in deep learning
- Disentangling feature and lazy training in deep neural networks
- Mean-field dynamics of infinite-dimensional particle systems: global shear versus random local forcing
- Universal characteristics of deep neural network loss surfaces from random matrix theory
- Black holes and the loss landscape in machine learning
- An introduction to machine learning: a perspective from statistical physics
- On weak ergodicity breaking in mean-field spin glasses
- Tradeoff of generalization error in unsupervised learning
- The role of the time-dependent Hessian in high-dimensional optimization
This page was built for publication: Comparing dynamics: deep neural networks versus glassy systems
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5854115)
