A Zeroth-Order Proximal Stochastic Gradient Method for Weakly Convex Stochastic Optimization
From MaRDI portal
Abstract: In this paper we analyze a zeroth-order proximal stochastic gradient method suitable for the minimization of weakly convex stochastic optimization problems. We consider nonsmooth and nonlinear stochastic composite problems, for which (sub-)gradient information might be unavailable. The proposed algorithm utilizes the well-known Gaussian smoothing technique, which yields unbiased zeroth-order gradient estimators of a related partially smooth surrogate problem (in which one of the two nonsmooth terms in the original problem's objective is replaced by a smooth approximation). This allows us to employ a standard proximal stochastic gradient scheme for the approximate solution of the surrogate problem, which is determined by a single smoothing parameter, and without the utilization of first-order information. We provide state-of-the-art convergence rates for the proposed zeroth-order method using minimal assumptions. The proposed scheme is numerically compared against alternative zeroth-order methods as well as a stochastic sub-gradient scheme on a standard phase retrieval problem. Further, we showcase the usefulness and effectiveness of our method for the unique setting of automated hyper-parameter tuning. In particular, we focus on automatically tuning the parameters of optimization algorithms by minimizing a novel heuristic model. The proposed approach is tested on a proximal alternating direction method of multipliers for the solution of -regularized PDE-constrained optimal control problems, with evident empirical success.
Recommendations
- A zeroth order method for stochastic weakly convex optimization
- The ``black-box optimization problem: zero-order accelerated stochastic method via kernel approximation
- Stochastic zeroth order descent with structured directions
- Zeroth-order nonconvex stochastic optimization: handling constraints, high dimensionality, and saddle points
- Zeroth-order algorithms for nonconvex-strongly-concave minimax problems with improved complexities
Cites work
- \(L\)-curve curvature bounds via Lanczos bidiagonalization
- A zeroth order method for stochastic weakly convex optimization
- Algorithm 866
- An efficient duality-based approach for PDE-constrained sparse optimization
- Convergence and regularization results for optimal control problems with sparsity functional
- Convergence of a random optimization method for constrained optimization problems
- Distributed optimization and statistical learning via the alternating direction method of multipliers
- Efficiency of minimizing compositions of convex functions and smooth maps
- Finding Optimal Algorithmic Parameters Using Derivative‐Free Optimization
- GCV for Tikhonov regularization by partial SVD
- Generalized ADMM with optimal indefinite proximal term for linearly constrained convex optimization
- Generalized Gradients and Applications
- scientific article; zbMATH DE number 3647643 (Why is no real title available?)
- scientific article; zbMATH DE number 5703572 (Why is no real title available?)
- scientific article; zbMATH DE number 1301898 (Why is no real title available?)
- scientific article; zbMATH DE number 6276119 (Why is no real title available?)
- IFISS: A Computational Laboratory for Investigating Incompressible Flow Problems
- Interior‐point methods and preconditioning for PDE‐constrained optimization problems involving sparsity terms
- Introduction to Derivative-Free Optimization
- Lectures on stochastic programming. Modeling and theory.
- Minimization by Random Search Techniques
- Multivariate stochastic approximation using a simultaneous perturbation gradient approximation
- Noisy zeroth-order optimization for non-smooth saddle point problems
- On the global and linear convergence of the generalized alternating direction method of multipliers
- Optimal Rates for Zero-Order Convex Optimization: The Power of Two Function Evaluations
- Optimization by Direct Search in Matrix Computations
- Pattern Search Methods for User-Provided Points: Application to Molecular Geometry Problems
- Phase retrieval: stability and recovery guarantees
- Proximité et dualité dans un espace hilbertien
- Random gradient-free minimization of convex functions
- Random optimization
- Residual whiteness principle for automatic parameter selection in _2-_2 image super-resolution problems
- Self-calibration and biconvex compressive sensing
- Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic Programming
- Stochastic model-based minimization of weakly convex functions
- Strong and Weak Convexity of Sets and Functions
- Variational Analysis
- Zeroth-order nonconvex stochastic optimization: handling constraints, high dimensionality, and saddle points
- Zeroth-order optimization with orthogonal random directions
- Zeroth-order stochastic compositional algorithms for risk-aware learning
Cited in
(6)- A zeroth order method for stochastic weakly convex optimization
- Derivative-free stochastic bilevel optimization for inverse problems
- Inexact zeroth-order nonsmooth and nonconvex stochastic composite optimization and applications
- Zeroth-order proximal clipped gradient method with shifts for distributed stochastic composite optimization problems with infinite variance
- An efficient active-set method with applications to sparse approximations and risk minimization
- A proximal Newton-type algorithm for zeroth-order stochastic composite optimization with a new norm test for sample size selection
This page was built for publication: A Zeroth-Order Proximal Stochastic Gradient Method for Weakly Convex Stochastic Optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6066421)
