Neural tangent kernel: convergence and generalization in neural networks (invited paper)
From MaRDI portal
(Redirected from Publication:6086982)
Abstract: At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We prove the positive-definiteness of the limiting NTK when the data is supported on the sphere and the non-linearity is non-polynomial. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.
Cited in
(24)- Doubly infinite residual neural networks: a diffusion process approach
- Convergence rates for shallow neural networks learned by gradient descent
- On the spectral bias of coupled frequency predictor-corrector triangular DNN: the convergence analysis
- Benign Overfitting and Noisy Features
- -Stable convergence of heavy-/light-tailed infinitely wide neural networks
- Learning physical models that can respect conservation laws
- Stochastic gradient descent: where optimization meets machine learning
- Learning with centered reproducing kernels
- Learning scattering waves via coupling physics-informed neural networks and their convergence analysis
- Accelerated zero-order SGD method for solving the black box optimization problem under ``overparametrization condition
- Homotopy relaxation training algorithms for infinite-width two-layer ReLU neural networks
- Large deviations of Gaussian neural networks with ReLu activation
- Conservative SPDEs as fluctuating mean field limits of stochastic gradient descent
- Overview frequency principle/spectral bias in deep learning
- Beyond the quadratic approximation: the multiscale structure of neural network loss landscapes
- Perturbational complexity by distribution mismatch: a systematic analysis of reinforcement learning in reproducing kernel Hilbert space
- A tighter generalization error bound for wide GCN based on loss landscape
- Rethinking cell-based neural architecture search: a theoretical perspective
- On the robustness of the minimim _2 interpolator
- A geometrical viewpoint on the benign overfitting property of the minimum _2-norm interpolant estimator and its universality
- VS-PINN: a fast and efficient training of physics-informed neural networks using variable-scaling methods for solving PDEs with stiff behavior
- Efficient identification of wide shallow neural networks with biases
- Trained quantum neural networks are Gaussian processes
- Stacked networks improve physics-informed training: applications to neural networks and deep operator networks
This page was built for publication: Neural tangent kernel: convergence and generalization in neural networks (invited paper)
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6086982)
