Follow the flow: Proximal flow inspired multi-step methods

arXiv2211.04653MaRDI QIDQ6416628FDOQ6416628

Yusheng Huang, Yi-Fan Sun

Publication date: 8 November 2022

Abstract: The gradient flow (GF) is an ODE for which its explicit Euler's discretization is the gradient descent method. In this work, we investigate a family of methods derived from emph{approximate implicit discretizations} of (GF), drawing the connection between larger stability regions and less sensitive hyperparameter tuning. We focus on the implicit $au$-step backwards differentiation formulas (BDFs), approximated in an inner loop with a few iterations of vanilla gradient descent, and give their convergence rate when the objective function is convex, strongly convex, or nonconvex. Numerical experiments show the wide range of effects of these different methods on extremely poorly conditioned problems, especially those brought about in training deep neural networks.

This page was built for publication: Follow the flow: Proximal flow inspired multi-step methods

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6416628)