Quadratic models for understanding neural network dynamics
From MaRDI portal
Publication:6399996
arXiv2205.11787MaRDI QIDQ6399996FDOQ6399996
Authors: Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin 
Publication date: 24 May 2022
Abstract: While neural networks can be approximated by linear models as their width increases, certain properties of wide neural networks cannot be captured by linear models. In this work we show that recently proposed Neural Quadratic Models can exhibit the "catapult phase" [Lewkowycz et al. 2020] that arises when training such models with large learning rates. We then empirically show that the behaviour of neural quadratic models parallels that of neural networks in generalization, especially in the catapult phase regime. Our analysis further demonstrates that quadratic models can be an effective tool for analysis of neural networks.
Has companion code repository: https://github.com/zhm1995/Neural-Quadratic-Model
This page was built for publication: Quadratic models for understanding neural network dynamics
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6399996)
