Eigenvalue-Corrected Natural Gradient Based on a New Approximation
From MaRDI portal
Abstract: Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factorization (EKFAC) (George et al., 2018), proposes an interpretation of viewing natural gradient update as a diagonal method, and corrects the inaccurate re-scaling factor in the Kronecker-factored eigenbasis. Gao et al. (2020) considers a new approximation to the natural gradient, which approximates the Fisher information matrix (FIM) to a constant multiplied by the Kronecker product of two matrices and keeps the trace equal before and after the approximation. In this work, we combine the ideas of these two methods and propose Trace-restricted Eigenvalue-corrected Kronecker Factorization (TEKFAC). The proposed method not only corrects the inexact re-scaling factor under the Kronecker-factored eigenbasis, but also considers the new approximation method and the effective damping technique proposed in Gao et al. (2020). We also discuss the differences and relationships among the Kronecker-factored approximations. Empirically, our method outperforms SGD with momentum, Adam, EKFAC and TKFAC on several DNNs.
Recommendations
- Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition
- Sketch-based empirical natural gradient methods for deep learning
- Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent
- Neural nets with a Newton conjugate gradient method on multiple GPUs
- Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*
Cites work
Cited in
(1)
This page was built for publication: Eigenvalue-Corrected Natural Gradient Based on a New Approximation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6077936)
