Combining resampling and reweighting for faithful stochastic optimization

DOI10.4310/CMS.2023.V21.N6.A6MaRDI QIDQ6071745zbMATH OpenOpenAlexFDO

Authors Jing An, Lexing Ying

Publication date 28 November 2023

Published in Communications in Mathematical Sciences (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2105.14694

zbMATH Keywords

non-convex optimizationresamplingreweightingstabilitystochastic asymptotics

Mathematics Subject Classification ID

Stochastic analysis in statistical mechanics (82M60)

Abstract: Many machine learning and data science tasks require solving non-convex optimization problems. When the loss function is a sum of multiple terms, a popular method is the stochastic gradient descent. Viewed as a process for sampling the loss function landscape, the stochastic gradient descent is known to prefer flat minima. Though this is desired for certain optimization problems such as in deep learning, it causes issues when the goal is to find the global minimum, especially if the global minimum resides in a sharp valley. Illustrated with a simple motivating example, we show that the fundamental reason is that the difference in the Lipschitz constants of multiple terms in the loss function causes stochastic gradient descent to experience different variances at different minima. In order to mitigate this effect and perform faithful optimization, we propose a combined resampling-reweighting scheme to balance the variance at local minima and extend to general loss functions. We explain from the numerical stability perspective how the proposed scheme is more likely to select the true global minimum, and the local convergence analysis perspective how it converges to a minimum faster when compared with the vanilla stochastic gradient descent. Experiments from robust statistics and computational chemistry are provided to demonstrate the theoretical findings.

Recommendations

• The inverse variance-flatness relation in stochastic gradient descent is critical for finding flat minima
• Stochastic gradient Hamiltonian Monte Carlo for non-convex learning
• Adaptive sampling strategies for stochastic optimization
• Globally convergent stochastic optimization with optimal asymptotic distribution
• Global convergence of stochastic gradient Hamiltonian Monte Carlo for nonconvex stochastic optimization: nonasymptotic performance bounds and momentum-based acceleration

Cited in

(1)

• Batched Stochastic Gradient Descent with Weighted Sampling

This page was built for publication: Combining resampling and reweighting for faithful stochastic optimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6071745)