Variance-based regularization with convex objectives
From MaRDI portal
Publication:5381122
Abstract: We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empirical likelihood, and we provide a number of finite-sample and asymptotic results characterizing the theoretical performance of the estimator. In particular, we show that our procedure comes with certificates of optimality, achieving (in some scenarios) faster rates of convergence than empirical risk minimization by virtue of automatically balancing bias and variance. We give corroborating empirical evidence showing that in practice, the estimator indeed trades between variance and absolute performance on a training sample, improving out-of-sample (test) performance over standard empirical risk minimization for a number of classification problems.
Recommendations
- Convergence rates of convex variational regularization
- Heuristic Parameter-Choice Rules for Convex Variational Regularization Based on Error Estimates
- Penalty-based smoothness conditions in convex variational regularization
- A range condition for polyconvex variational regularization
- Stochastic variance-reduced cubic regularization methods
- Convex regularization in statistical inverse learning problems
- Variational regularization in inverse problems and machine learning
- Regularization with non-convex separable constraints
- Continuous regularized variable-metric proximal minimization method
- Theory and examples of variational regularization with non-metric fitting functionals
Cites work
- scientific article; zbMATH DE number 439380 (Why is no real title available?)
- scientific article; zbMATH DE number 5654889 (Why is no real title available?)
- scientific article; zbMATH DE number 49190 (Why is no real title available?)
- scientific article; zbMATH DE number 1220667 (Why is no real title available?)
- scientific article; zbMATH DE number 1332320 (Why is no real title available?)
- scientific article; zbMATH DE number 2034517 (Why is no real title available?)
- scientific article; zbMATH DE number 3446442 (Why is no real title available?)
- scientific article; zbMATH DE number 2107836 (Why is no real title available?)
- 10.1162/153244303321897690
- 10.1162/1532443041424337
- A Bennett concentration inequality and its application to suprema of empirical processes
- A result of Vapnik with applications
- An introduction to support vector machines and other kernel-based learning methods.
- Capacity of reproducing kernel spaces in learning theory
- Concentration inequalities. A nonasymptotic theory of independence
- Convexity, Classification, and Risk Bounds
- Covering numbers of Gaussian reproducing kernel Hilbert spaces
- Empirical likelihood
- Empirical likelihood ratio confidence regions
- Introduction to nonparametric estimation
- Learning without concentration
- Lectures on Stochastic Programming
- Local Rademacher complexities
- Local Rademacher complexities and oracle inequalities in risk minimization. (2004 IMS Medallion Lecture). (With discussions and rejoinder)
- On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
- Optimal aggregation of classifiers in statistical learning.
- Oracle-based robust optimization via online learning
- Regularization and Variable Selection Via the Elastic Net
- Robust optimization
- Smooth discrimination analysis
- Smoothing spline ANOVA models
- Statistics for high-dimensional data. Methods, theory and applications.
- Statistics of robust optimization: a generalized empirical likelihood approach
- Theory of Classification: a Survey of Some Recent Advances
- Uniform Central Limit Theorems
- Weak convergence and empirical processes. With applications to statistics
Cited in
(20)- Weakly-convex-concave min-max optimization: provable algorithms and applications in machine learning
- Statistics of robust optimization: a generalized empirical likelihood approach
- Robust and distributionally robust optimization models for linear support vector machine
- Convergence rates of convex variational regularization
- Learning models with uniform performance via distributionally robust optimization
- Robust Simulation with Likelihood-Ratio Constrained Input Uncertainty
- Toward theoretical understandings of robust Markov decision processes: sample complexity and asymptotics
- Variance regularization in sequential Bayesian optimization
- A survey of learning criteria going beyond the usual risk
- Sample average approximation with heavier tails II: localization in stochastic convex optimization and persistence results for the Lasso
- Regularization via mass transportation
- Enhanced Balancing of Bias-Variance Tradeoff in Stochastic Estimation: A Minimax Perspective
- Distributionally robust optimization. A review on theory and applications
- scientific article; zbMATH DE number 7370573 (Why is no real title available?)
- Distributionally robust bottleneck combinatorial problems: uncertainty quantification and robust decision making
- General procedure to provide high-probability guarantees for stochastic saddle point problems
- Conditional variance penalties and domain shift robustness
- Learning with risks based on M-location
- Heuristic Parameter-Choice Rules for Convex Variational Regularization Based on Error Estimates
- Coefficient-based regularization network with variance loss for error
This page was built for publication: Variance-based regularization with convex objectives
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5381122)
