QNG: a quasi-natural gradient method for large-scale statistical learning

DOI10.1137/20M1376753MaRDI QIDQ5067429zbMATH OpenOpenAlexFDO

Authors Xiaoyu He, Zibin Zheng, Yuren Zhou, Chuan Chen

Publication date 1 April 2022

Published in SIAM Journal on Optimization (Search for Journal in Brave)

Full work available at URL https://doi.org/10.1137/20m1376753

zbMATH Keywords

Fisher information matrixlarge-scale stochastic optimizationnatural gradientstatistical learning

Mathematics Subject Classification ID

Numerical mathematical programming methods (65K05) Large-scale problems in mathematical programming (90C06) Stochastic programming (90C15) Methods of quasi-Newton type (90C53)

Recommendations

• SGD-QN: careful quasi-Newton stochastic gradient descent
• A stochastic quasi-Newton method for large-scale optimization
• Statistical learning by natural gradient descent
• Quasi-Newton method: a new direction
• Global convergence of online limited memory BFGS

Cites work

• A Stochastic Approximation Method
• A stochastic quasi-Newton method for large-scale optimization
• Benchmarking optimization software with performance profiles.
• Global convergence of online limited memory BFGS
• scientific article; zbMATH DE number 7306852 (Why is no real title available?)
• scientific article; zbMATH DE number 5060482 (Why is no real title available?)
• Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization
• Machine learning. A probabilistic perspective
• On the limited memory BFGS method for large scale optimization
• Optimization
• Optimization methods for large-scale machine learning
• Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization
• Updating Quasi-Newton Matrices with Limited Storage

Cited in

(4)

• Sketch-based empirical natural gradient methods for deep learning
• Understanding approximate Fisher information for fast convergence of natural gradient descent in wide neural networks*
• Quasi-Newton methods for machine learning: forget the past, just sample
• A distributed optimisation framework combining natural gradient with Hessian-free for discriminative sequence training

Describes a project that uses

Uses Software

• PMTK
• L-BFGS

This page was built for publication: QNG: a quasi-natural gradient method for large-scale statistical learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5067429)