GADMM: fast and communication efficient framework for distributed machine learning
From MaRDI portal
Abstract: When the data is distributed across multiple servers, lowering the communication cost between the servers (or workers) while solving the distributed learning problem is an important problem and is the focus of this paper. In particular, we propose a fast, and communication-efficient decentralized framework to solve the distributed machine learning (DML) problem. The proposed algorithm, Group Alternating Direction Method of Multipliers (GADMM) is based on the Alternating Direction Method of Multipliers (ADMM) framework. The key novelty in GADMM is that it solves the problem in a decentralized topology where at most half of the workers are competing for the limited communication resources at any given time. Moreover, each worker exchanges the locally trained model only with two neighboring workers, thereby training a global model with a lower amount of communication overhead in each exchange. We prove that GADMM converges to the optimal solution for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication-efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets. Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove its convergence under the time-varying network topology of the workers.
Recommendations
- A general distributed dual coordinate optimization framework for regularized loss minimization
- Communication-efficient distributed optimization of self-concordant empirical loss
- An efficient distributed learning algorithm based on effective local functional approximations
- DSA: decentralized double stochastic averaging gradient algorithm
- scientific article; zbMATH DE number 6982986
Cites work
- A Convergent Incremental Gradient Method with a Constant Step Size
- A dual algorithm for the solution of nonlinear variational problems via finite element approximation
- A Proximal Gradient Algorithm for Decentralized Composite Optimization
- Achieving Geometric Convergence for Distributed Optimization Over Time-Varying Graphs
- Asynchronous Saddle Point Algorithm for Stochastic Optimization in Heterogeneous Networks
- Communication-Censored ADMM for Decentralized Consensus Optimization
- Distributed Constrained Optimization by Consensus-Based Primal-Dual Perturbation Method
- Distributed optimization and statistical learning via the alternating direction method of multipliers
- Distributed Optimization Over Time-Varying Directed Graphs
- Distributed Subgradient Methods for Multi-Agent Optimization
- Dual Averaging for Distributed Optimization: Convergence Analysis and Network Scaling
- Fast Distributed Gradient Methods
- scientific article; zbMATH DE number 3574917 (Why is no real title available?)
- Minimizing finite sums with the stochastic average gradient
- Multi-Agent Distributed Optimization via Inexact Consensus ADMM
- On the capacity of channels with Gaussian and non-Gaussian noise
- On the Convergence Rate of Incremental Aggregated Gradient Algorithms
- Parallel multi-block ADMM with \(o(1/k)\) convergence
- Proximity Without Consensus in Online Multiagent Optimization
- Some Simple Applications of the Travelling Salesman Problem
- The N-City Travelling Salesman Problem: Statistical Mechanics and the Metropolis Algorithm
- The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent
Cited in
(11)- EGC: entropy-based gradient compression for distributed deep learning
- Communication-efficient distributed \(M\)-estimation with missing data
- Distributed Deep Learning on Heterogeneous Computing Resources Using Gossip Communication
- Layer-wise adaptive gradient sparsification for distributed deep learning with convergence guarantees
- Efficient and reliable overlay networks for decentralized federated learning
- Decentralized learning for wireless communications and networking
- Distributed quantile regression in decentralized optimization
- Distributed optimal subsampling for quantile regression with massive data
- Federated Offline Reinforcement Learning
- Distributed optimization for penalized regression in massive compositional data
- Byzantine-robust distributed vertical learning over time-varying networks
This page was built for publication: GADMM: fast and communication efficient framework for distributed machine learning
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4969135)
