Diametrical risk minimization: theory and computations

DOI10.1007/S10994-021-06036-0MaRDI QIDQ6134352zbMATH OpenOpenAlexFDO

Authors Matthew D. Norton, Johannes O. Royset

Publication date 22 August 2023

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1910.10844

zbMATH Keywords

empirical risk minimizationgeneralization errorsolution stability

Mathematics Subject Classification ID

Analysis of algorithms (68W40) Statistical methods; risk measures (91G70)

Abstract: The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.

Recommendations

• Stability properties of empirical risk minimization over Donsker classes
• Risk bounds for statistical learning
• Learning Theory
• Minimax rate of convergence and the performance of empirical risk minimization in phase recovery
• Empirical risk minimization for heavy-tailed losses

Cites work

• 10.1162/153244302760200704
• An Optimization Primer
• Applications of convex optimization in signal processing and digital communication
• Characterization of the equivalence of robustification and regularization in linear and matrix regression
• Concentration inequalities. A nonasymptotic theory of independence
• Distributionally Robust Convex Optimization
• Entropy-SGD: biasing gradient descent into wide valleys
• Fabrication-adaptive optimization with an application to photonic crystal design
• Flat Minima
• Learnability, stability and uniform convergence
• Lectures on Stochastic Programming
• Lipschitz behavior of the robust regularization
• Multivariate Nonnegative Quadratic Mappings
• On robust solutions to linear least squares problems affected by data uncertainty and implementation errors with application to stochastic signal modeling
• On sample size control in sample average approximations for solving smooth stochastic programs
• Optimal budget allocation for sample average approximation
• Quantitative stability analysis for distributionally robust optimization with moment constraints
• Regularisation of neural networks by enforcing Lipschitz continuity
• Robust convex optimization
• Robust optimization using computer experiments
• Robust sample average approximation
• Sample average approximation with heavier tails. I: Non-asymptotic bounds with weak assumptions and stochastic constraints
• Sample average approximation with sparsity-inducing penalty for high-dimensional stochastic programming
• Statistics of robust optimization: a generalized empirical likelihood approach
• Variational Analysis
• Variational theory for optimization under stochastic ambiguity

Cited in

(2)

• Risk-adaptive approaches to stochastic optimization: a survey
• Minimax risk over hyperrectangles, and implications

This page was built for publication: Diametrical risk minimization: theory and computations

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6134352)