Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems
From MaRDI portal
Abstract: Stochastic gradient methods (SGMs) are predominant approaches for solving stochastic optimization. On smooth nonconvex problems, a few acceleration techniques have been applied to improve the convergence rate of SGMs. However, little exploration has been made on applying a certain acceleration technique to a stochastic subgradient method (SsGM) for nonsmooth nonconvex problems. In addition, few efforts have been made to analyze an (accelerated) SsGM with delayed derivatives. The information delay naturally happens in a distributed system, where computing workers do not coordinate with each other. In this paper, we propose an inertial proximal SsGM for solving nonsmooth nonconvex stochastic optimization problems. The proposed method can have guaranteed convergence even with delayed derivative information in a distributed environment. Convergence rate results are established to three classes of nonconvex problems: weakly-convex nonsmooth problems with a convex regularizer, composite nonconvex problems with a nonsmooth convex regularizer, and smooth nonconvex problems. For each problem class, the convergence rate is in the expected value of the gradient norm square, for iterations. In a distributed environment, the convergence rate of the proposed method will be slowed down by the information delay. Nevertheless, the slow-down effect will decay with the number of iterations for the latter two problem classes. We test the proposed method on three applications. The numerical results clearly demonstrate the advantages of using the inertial-based acceleration. Furthermore, we observe higher parallelization speed-up in asynchronous updates over the synchronous counterpart, though the former uses delayed derivatives. Our source code is released at https://github.com/RPI-OPT/Inertial-SsGM
Recommendations
- Accelerated gradient methods for nonconvex nonlinear and stochastic programming
- Parallel and distributed asynchronous adaptive stochastic gradient methods
- Distributed stochastic gradient tracking methods with momentum acceleration for non-convex optimization
- Proximally guided stochastic subgradient method for nonsmooth, nonconvex problems
- Distributed stochastic optimization with large delays
Cites work
- A hybrid stochastic optimization framework for composite nonconvex optimization
- A Stochastic Approximation Method
- Accelerated gradient methods for nonconvex nonlinear and stochastic programming
- Adaptive subgradient methods for online learning and stochastic optimization
- Algorithms of inertial mirror descent in convex problems of stochastic optimization
- An Incremental Gradient(-Projection) Method with Momentum Term and Adaptive Stepsize Rule
- An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping
- Convergence of a splitting inertial proximal method for monotone operators
- Distributed asynchronous incremental subgradient methods
- Efficiency of minimizing compositions of convex functions and smooth maps
- Heavy-ball method in nonconvex optimization problems
- scientific article; zbMATH DE number 3850830 (Why is no real title available?)
- scientific article; zbMATH DE number 4164577 (Why is no real title available?)
- iPiano: inertial proximal algorithm for nonconvex optimization
- iPiasco: inertial proximal algorithm for strongly convex optimization
- Local convergence of the heavy-ball method and iPiano for non-convex optimization
- Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods
- On Distributed Nonconvex Optimization: Projected Subgradient Method for Weakly Convex Problems in Networks
- On the convergence of asynchronous parallel iteration with unbounded delays
- On unbounded delays in asynchronous parallel fixed-point algorithms
- Phase retrieval: stability and recovery guarantees
- Proximité et dualité dans un espace hilbertien
- Rank-Sparsity Incoherence for Matrix Decomposition
- Robust principal component analysis?
- Robust Stochastic Approximation Approach to Stochastic Programming
- Solving (most) of a set of quadratic equalities: composite optimization for robust phase retrieval
- Some methods of speeding up the convergence of iteration methods
- Stochastic model-based minimization of weakly convex functions
- The nonsmooth landscape of phase retrieval
- Weak Convergence of a Relaxed and Inertial Hybrid Projection-Proximal Point Algorithm for Maximal Monotone Operators in Hilbert Space
Cited in
(3)
This page was built for publication: Distributed stochastic inertial-accelerated methods with delayed derivatives for nonconvex problems
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5863523)
