Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems

DOI10.1109/TSP.2021.3097211MaRDI QIDQ6353421arXivFDO

Authors Zhiguo Wang, Jiawei Zhang, Tsung-Hui Chang, Jian Li, Zhi-Quan Luo

Publication date 10 November 2020

Published in arXiv

Full work available at URL https://doi.org/10.1109/TSP.2021.3097211

Mathematics Subject Classification ID

Signal theory (characterization, reconstruction, filtering, etc.) (94A12)

arXiv classification

cs.LGmath.OC

Abstract: While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper presents a new (stochastic) distributed algorithm with Nesterov momentum for accelerated optimization of non-convex and non-smooth problems. Theoretically, we show that the proposed algorithm can achieve an $epsilon$-stationary solution under a constant step size with $mathcalO(1/epsilo{n}^2)$ computation complexity and $mathcalO(1/epsilon)$ communication complexity. When compared to the existing gradient tracking based methods, the proposed algorithm has the same order of computation complexity but lower order of communication complexity. To the best of our knowledge, the presented result is the first stochastic algorithm with the $mathcalO(1/epsilon)$ communication complexity for non-convex and non-smooth problems. Numerical experiments for a distributed non-convex regression problem and a deep neural network based classification problem are presented to illustrate the effectiveness of the proposed algorithms.

This page was built for publication: Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6353421)