Randomized smoothing for stochastic optimization
From MaRDI portal
Abstract: We analyze convergence rates of stochastic optimization procedures for non-smooth convex optimization problems. By combining randomized smoothing techniques with accelerated gradient methods, we obtain convergence rates of stochastic optimization procedures, both in expectation and with high probability, that have optimal dependence on the variance of the gradient estimates. To the best of our knowledge, these are the first variance-based rates for non-smooth optimization. We give several applications of our results to statistical estimation problems, and provide experimental results that demonstrate the effectiveness of the proposed algorithms. We also describe how a combination of our algorithm with recent work on decentralized optimization yields a distributed stochastic optimization algorithm that is order-optimal.
The convergence rates of stochastic optimization algorithms for nonsmooth convex optimization problems are analyzed. The authors also discuss several applications of the proposed results to statistical estimation problems. Some experimental results are provided to demonstrate the effectiveness of the proposed algorithms.
Recommendations
- Randomized smoothing variance reduction method for large-scale non-smooth convex optimization
- A stochastic Nesterov's smoothing accelerated method for general nonsmooth constrained stochastic composite convex optimization
- A smoothing stochastic gradient method for composite optimization
- Accelerated randomized stochastic optimization.
- Gradient-free proximal methods with inexact oracle for convex stochastic nonsmooth optimization problems on the simplex
Cited in
(45)- Supervised nonnegative matrix factorization via minimization of regularized Moreau-envelope of divergence function with application to music transcription
- Stochastic mirror descent method for distributed multi-agent optimization
- On smoothing, regularization, and averaging in stochastic approximation methods for stochastic variational inequality problems
- On the convergence rate issues of general Markov search for global minimum
- Gradient-free method for nonsmooth distributed optimization
- Randomized smoothing variance reduction method for large-scale non-smooth convex optimization
- A stochastic Nesterov's smoothing accelerated method for general nonsmooth constrained stochastic composite convex optimization
- Laplacian smoothing gradient descent
- Algorithms for stochastic optimization with function or expectation constraints
- Incremental gradient-free method for nonsmooth distributed optimization
- Sample average approximations of strongly convex stochastic programs in Hilbert spaces
- Complexity guarantees for an implicit smoothing-enabled method for stochastic MPECs
- A smoothing stochastic gradient method for composite optimization
- A Stochastic Smoothing Algorithm for Semidefinite Programming
- Penalty methods with stochastic approximation for stochastic nonlinear programming
- scientific article; zbMATH DE number 1748526 (Why is no real title available?)
- A sparsity preserving stochastic gradient methods for sparse regression
- Rejoinder
- Lower bounds for parallel and randomized convex optimization
- Bias reduction in sample-based optimization
- Optimal convergence rates for convex distributed optimization in networks
- An inexact variable metric proximal point algorithm for generic quasi-Newton acceleration
- Smoothed functional algorithms for stochastic optimization using q-Gaussian distributions
- Extragradient Method with Variance Reduction for Stochastic Variational Inequalities
- A dual approach for optimal algorithms in distributed optimization over networks
- The impact of randomization in smoothing networks
- Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization
- Gradient-free federated learning methods with l₁ and l₂-randomization for non-smooth convex stochastic optimization problems
- An Improved Unconstrained Approach for Bilevel Optimization
- Short paper -- A note on the Frank-Wolfe algorithm for a class of nonconvex and nonsmooth optimization problems
- Algorithms with gradient clipping for stochastic optimization with heavy-tailed noise
- Truncated Cauchy random perturbations for smoothed functional-based stochastic optimization
- Convergence properties of stochastic proximal subgradient method in solving a class of composite optimization problems with cardinality regularizer
- Leveraging randomized smoothing for optimal control of nonsmooth dynamical systems
- A new computational framework for log-concave density estimation
- Optimality and duality for nonconvex fuzzy optimization using granular differentiability method
- Stochastic zeroth order descent with structured directions
- Nonsmooth projection-free optimization with functional constraints
- Optimization on a finer scale: bounded local subgradient variation perspective
- Inexact zeroth-order nonsmooth and nonconvex stochastic composite optimization and applications
- Central limit theorems for vector-valued composite functionals with smoothing and applications
- Zero order algorithm for decentralized optimization problems
- Computationally efficient and statistically optimal robust high-dimensional linear regression
- Zeroth-order random subspace algorithm for non-smooth convex optimization
- A unified mini-batch stochastic accelerated method for nonconvex stochastic programming
This page was built for publication: Randomized smoothing for stochastic optimization
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2910890)
