The error-feedback framework: SGD with delayed gradients

MaRDI QIDQ5149264FDOQ5149264

Authors: S. U. Stich, Sai Praneeth Karimireddy 

Publication date: 8 February 2021

Full work available at URL: https://arxiv.org/abs/1909.05350

Recommendations

• Distributed stochastic optimization with large delays
• A sharp convergence rate for a model equation of the asynchronous stochastic gradient descent
• A distributed flexible delay-tolerant proximal gradient algorithm
• Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems
• Stochastic gradient descent with Polyak's learning rate

zbMATH Keywords

optimizationmachine learningstochastic gradient descentgradient compressionerror-feedbackdelayed gradientserror-compensationlocal SGD

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05)

Cites Work

• Introductory lectures on convex optimization. A basic course.
• A Stochastic Approximation Method
• Robust Stochastic Approximation Approach to Stochastic Programming
• Title not available (Why is that?)
• Title not available (Why is that?)
• Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework
• Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
• Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm
• Cubic regularization of Newton method and its global performance
• Large-scale machine learning with stochastic gradient descent
• Optimal distributed online prediction using mini-batches
• Linear convergence of first order methods for non-strongly convex optimization
• New method of stochastic approximation type
• Optimization methods for large-scale machine learning
• Communication-efficient algorithms for statistical optimization
• Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression
• Parallelizing stochastic gradient descent for least squares regression: mini-batching, averaging, and model misspecification
• An Asynchronous Mini-Batch Algorithm for Regularized Stochastic Optimization
• Perturbed iterate analysis for asynchronous stochastic optimization
• Gradient descent learns linear dynamical systems
• Improved asynchronous parallel optimization analysis for stochastic incremental methods

Cited In (3)

• Efficient and reliable overlay networks for decentralized federated learning
• Non-ergodic linear convergence property of the delayed gradient descent under the strongly convexity and the Polyak-Łojasiewicz condition
• Faster Rates for Compressed Federated Learning with Client-Variance Reduction

Uses Software

• TernGrad
• HOGWILD
• SCAFFOLD

This page was built for publication: The error-feedback framework: SGD with delayed gradients

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5149264)