Variance-based regularization with convex objectives
From MaRDI portal
Publication:5381122
zbMATH Open1489.62193arXiv1610.02581MaRDI QIDQ5381122FDOQ5381122
Authors: John C. Duchi, Hongseok Namkoong 
Publication date: 7 June 2019
Abstract: We develop an approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error. Our approach builds off of techniques for distributionally robust optimization and Owen's empirical likelihood, and we provide a number of finite-sample and asymptotic results characterizing the theoretical performance of the estimator. In particular, we show that our procedure comes with certificates of optimality, achieving (in some scenarios) faster rates of convergence than empirical risk minimization by virtue of automatically balancing bias and variance. We give corroborating empirical evidence showing that in practice, the estimator indeed trades between variance and absolute performance on a training sample, improving out-of-sample (test) performance over standard empirical risk minimization for a number of classification problems.
Full work available at URL: https://arxiv.org/abs/1610.02581
Recommendations
- Convergence rates of convex variational regularization
- Heuristic Parameter-Choice Rules for Convex Variational Regularization Based on Error Estimates
- Penalty-based smoothness conditions in convex variational regularization
- A range condition for polyconvex variational regularization
- Stochastic variance-reduced cubic regularization methods
- Convex regularization in statistical inverse learning problems
- Variational regularization in inverse problems and machine learning
- Regularization with non-convex separable constraints
- Continuous regularized variable-metric proximal minimization method
- Theory and examples of variational regularization with non-metric fitting functionals
Nonparametric estimation (62G05) General nonlinear regression (62J02) Asymptotic distribution theory in statistics (62E20)
Cites Work
- Smoothing spline ANOVA models
- Weak convergence and empirical processes. With applications to statistics
- Title not available (Why is that?)
- Statistics for high-dimensional data. Methods, theory and applications.
- Empirical likelihood ratio confidence regions
- Title not available (Why is that?)
- Regularization and Variable Selection Via the Elastic Net
- An introduction to support vector machines and other kernel-based learning methods.
- Empirical likelihood
- Title not available (Why is that?)
- Title not available (Why is that?)
- Introduction to nonparametric estimation
- Title not available (Why is that?)
- On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities
- Title not available (Why is that?)
- Smooth discrimination analysis
- A Bennett concentration inequality and its application to suprema of empirical processes
- Local Rademacher complexities and oracle inequalities in risk minimization. (2004 IMS Medallion Lecture). (With discussions and rejoinder)
- Local Rademacher complexities
- Uniform Central Limit Theorems
- Robust optimization
- 10.1162/153244303321897690
- Convexity, Classification, and Risk Bounds
- Title not available (Why is that?)
- Lectures on Stochastic Programming
- Optimal aggregation of classifiers in statistical learning.
- Learning without concentration
- Title not available (Why is that?)
- 10.1162/1532443041424337
- Theory of Classification: a Survey of Some Recent Advances
- Capacity of reproducing kernel spaces in learning theory
- Title not available (Why is that?)
- A result of Vapnik with applications
- Covering numbers of Gaussian reproducing kernel Hilbert spaces
- Oracle-Based Robust Optimization via Online Learning
- Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach
Cited In (18)
- Learning models with uniform performance via distributionally robust optimization
- Distributionally robust bottleneck combinatorial problems: uncertainty quantification and robust decision making
- Heuristic Parameter-Choice Rules for Convex Variational Regularization Based on Error Estimates
- General procedure to provide high-probability guarantees for stochastic saddle point problems
- Toward theoretical understandings of robust Markov decision processes: sample complexity and asymptotics
- Conditional variance penalties and domain shift robustness
- Sample average approximation with heavier tails II: localization in stochastic convex optimization and persistence results for the Lasso
- Robust Simulation with Likelihood-Ratio Constrained Input Uncertainty
- Title not available (Why is that?)
- Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach
- Weakly-convex–concave min–max optimization: provable algorithms and applications in machine learning
- Learning with risks based on M-location
- Coefficient-based regularization network with variance loss for error
- Variance Regularization in Sequential Bayesian Optimization
- Distributionally robust optimization. A review on theory and applications
- A survey of learning criteria going beyond the usual risk
- Convergence rates of convex variational regularization
- Robust and distributionally robust optimization models for linear support vector machine
Uses Software
This page was built for publication: Variance-based regularization with convex objectives
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5381122)
