Weighted neural tangent kernel: a generalized and improved network-induced kernel

DOI10.1007/S10994-023-06356-3MaRDI QIDQ6134348zbMATH OpenFDO

Authors

Publication date 22 August 2023

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2103.11558

zbMATH Keywords

adjusted descent directionneural tangent kernelover-parameterization

Mathematics Subject Classification ID

Numerical optimization and variational techniques (65K10) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: The Neural Tangent Kernel (NTK) has recently attracted intense study, as it describes the evolution of an over-parameterized Neural Network (NN) trained by gradient descent. However, it is now well-known that gradient descent is not always a good optimizer for NNs, which can partially explain the unsatisfactory practical performance of the NTK regression estimator. In this paper, we introduce the Weighted Neural Tangent Kernel (WNTK), a generalized and improved tool, which can capture an over-parameterized NN's training dynamics under different optimizers. Theoretically, in the infinite-width limit, we prove: i) the stability of the WNTK at initialization and during training, and ii) the equivalence between the WNTK regression estimator and the corresponding NN estimator with different learning rates on different parameters. With the proposed weight update algorithm, both empirical and analytical WNTKs outperform the corresponding NTKs in numerical experiments.

Recommendations

• A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics
• Overparameterization and generalization error: weighted trigonometric interpolation
• Gradients weights improve regression and classification
• Generalization error of minimum weighted norm and kernel interpolation
• Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

Cites work

• Do we need hundreds of classifiers to solve real world classification problems?
• scientific article; zbMATH DE number 6378127 (Why is no real title available?)
• scientific article; zbMATH DE number 2050349 (Why is no real title available?)
• scientific article; zbMATH DE number 6253909 (Why is no real title available?)
• Large scale multiple kernel learning
• Learning data-adaptive non-parametric kernels
• Linearized two-layers neural networks in high dimension
• Multiple kernel learning algorithms
• Products of many large random matrices and gradients in deep neural networks
• Wide neural networks of any depth evolve as linear models under gradient descent ^*

This page was built for publication: Weighted neural tangent kernel: a generalized and improved network-induced kernel

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6134348)