Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters

MaRDI QIDQ4558160zbMATH OpenFDO

Authors

Publication date 21 November 2018

Full work available at URL https://arxiv.org/abs/1704.03540, http://jmlr.csail.mit.edu/papers/v19/17-444.html

zbMATH Keywords

machine learningconvergencedistributed systemproximal gradientmodel parallelpartially asynchronous

Mathematics Subject Classification ID

Computational learning theory (68Q32) Nonlinear programming (90C30) Convergence and divergence of series and sequences (40A05) Parallel algorithms in computer science (68W10) Distributed systems (68M14)

Abstract: With ever growing data volume and model size, an error-tolerant, communication efficient, yet versatile distributed algorithm has become vital for the success of many large-scale machine learning applications. In this work we propose m-PAPG, an implementation of the flexible proximal gradient algorithm in model parallel systems equipped with the partially asynchronous communication protocol. The worker machines communicate asynchronously with a controlled staleness bound $s$ and operate at different frequencies. We characterize various convergence properties of m-PAPG: 1) Under a general non-smooth and non-convex setting, we prove that every limit point of the sequence generated by m-PAPG is a critical point of the objective function; 2) Under an error bound condition, we prove that the function value decays linearly for every $s$ steps; 3) Under the Kurdyka-$L$ojasiewicz inequality, we prove that the sequences generated by m-PAPG converge to the same critical point, provided that a proximal Lipschitz condition is satisfied.

Recommendations

• Asynchronous stochastic coordinate descent: parallelism and convergence properties
• Asynchronous parallel algorithms for nonconvex optimization
• A distributed flexible delay-tolerant proximal gradient algorithm
• Distributed asynchronous incremental subgradient methods
• Convergence Analysis of the Proximal Gradient Method in the Presence of the Kurdyka–Łojasiewicz Property Without Global Lipschitz Assumptions
• Proximal-proximal-gradient method
• Proximal gradient algorithms under local Lipschitz gradient continuity. A convergence and robustness analysis of PANOC
• Random Coordinate Descent Methods for Nonseparable Composite Optimization
• Composite optimization with coupling constraints via dual proximal gradient method with applications to asynchronous networks
• Distributed composite optimization for multi-agent systems with asynchrony

Cited in

(8)

• Composite optimization with coupling constraints via dual proximal gradient method with applications to asynchronous networks
• Asynchronous ADMM for nonlinear continuous-time systems
• A distributed flexible delay-tolerant proximal gradient algorithm
• Asynchronous fully-decentralized SGD in the cluster-based model
• Sharper Bounds for Proximal Gradient Algorithms with Errors
• An accelerated proximal algorithm for regularized nonconvex and nonsmooth bi-level optimization
• Primal-dual algorithms for multi-agent structured optimization over message-passing architectures with bounded communication delays
• Distributed asynchronous incremental subgradient methods

This page was built for publication: Distributed Proximal Gradient Algorithm for Partially Asynchronous Computer Clusters

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4558160)