Learning models with uniform performance via distributionally robust optimization

DOI10.1214/20-AOS2004zbMATH Open1473.62019arXiv1810.08750OpenAlexW3188960136MaRDI QIDQ820804FDOQ820804

John C. Duchi, Hongseok Namkoong

Publication date: 28 September 2021

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: A common goal in statistics and machine learning is to learn models that can perform well against distributional shifts, such as latent heterogeneous subpopulations, unknown covariate shifts, or unmodeled temporal effects. We develop and analyze a distributionally robust stochastic optimization (DRO) framework that learns a model providing good performance against perturbations to the data-generating distribution. We give a convex formulation for the problem, providing several convergence guarantees. We prove finite-sample minimax upper and lower bounds, showing that distributional robustness sometimes comes at a cost in convergence rates. We give limit theorems for the learned parameters, where we fully specify the limiting distribution so that confidence intervals can be computed. On real tasks including generalizing to unknown subpopulations, fine-grained recognition, and providing good tail performance, the distributionally robust approach often exhibits improved performance.

Full work available at URL: https://arxiv.org/abs/1810.08750

Recommendations

• scientific article; zbMATH DE number 7668294
• A robust learning approach for regression models based on distributionally robust optimization
• A stochastic subgradient method for distributionally robust non-convex and non-smooth learning
• Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization
• Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations

zbMATH Keywords

minimax optimalityrobust optimizationrisk-averse learning

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Learning and adaptive systems in artificial intelligence (68T05) Minimax procedures in statistical decision theory (62C20)

Cites Work

• Weak convergence and empirical processes. With applications to statistics
• Finite mixture models
• Title not available (Why is that?)
• Empirical likelihood ratio confidence regions
• Coherent measures of risk
• Title not available (Why is that?)
• Variational Analysis
• Regularization and Variable Selection Via the Elastic Net
• Robust Statistics
• Asymptotics in statistics. Some basic concepts.
• A data-driven software tool for enabling cooperative information sharing among police departments
• Inference in hidden Markov models.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Asymptotic Statistics
• Robust Statistics
• Statistical estimation in varying coefficient models
• Efficient Estimation and Inferences for Varying-Coefficient Models
• Title not available (Why is that?)
• Improving predictive inference under covariate shift by weighting the log-likelihood function
• Classifier technology and the illusion of progress
• Local Rademacher complexities
• Concentration inequalities. A nonasymptotic theory of independence
• Title not available (Why is that?)
• The Integral of a Symmetric Unimodal Function over a Symmetric Convex Set and Some Probability Inequalities
• Stochastic orders
• Title not available (Why is that?)
• The Brunn-Minkowski inequality
• Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems
• Lectures on Stochastic Programming
• Higher moment coherent risk measures
• Title not available (Why is that?)
• Title not available (Why is that?)
• Robustness
• Rényi Divergence and Kullback-Leibler Divergence
• Data-driven chance constrained stochastic program
• A new look at independence
• Epi‐consistency of convex stochastic programs
• Distributionally Robust Stochastic Programming
• Maximin effects in inhomogeneous large-scale data
• On Kusuoka Representation of Law Invariant Risk Measures
• A theory of learning from different domains
• Statistical decision functions which minimize the maximum risk
• Data-driven robust optimization
• Ambiguity in portfolio selection
• Title not available (Why is that?)
• Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization
• Linear Minimax Regret Estimation of Deterministic Parameters with Bounded Data Uncertainties
• Robust risk measurement and model risk
• Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations
• Minimax optimal control of stochastic uncertain systems with relative entropy constraints
• The empirical likelihood approach to quantifying uncertainty in sample average approximation
• A framework for optimization under ambiguity
• Entropic value-at-risk: a new coherent risk measure
• Robust empirical optimization is almost the same as mean-variance optimization
• Path-space information bounds for uncertainty quantification and sensitivity analysis of stochastic dynamics
• Robust sensitivity analysis for stochastic systems
• Confidence Intervals for Maximin Effects in Inhomogeneous Large-Scale Data
• Robust Bounds on Risk-Sensitive Functionals via Rényi Divergence
• Robust Analysis in Stochastic Simulation: Computation and Performance Guarantees
• Title not available (Why is that?)
• Quantifying Distributional Model Risk via Optimal Transport
• Robust Wasserstein profile inference and applications to machine learning
• Variance-based regularization with convex objectives

Cited In (18)

• Title not available (Why is that?)
• Distributionally robust optimization with moment ambiguity sets
• Toward theoretical understandings of robust Markov decision processes: sample complexity and asymptotics
• Regularized distributionally robust optimization with application to the index tracking problem
• Statistical inference for generative adversarial networks and other minimax problems
• Robust inference for moment condition models without rational expectations
• Title not available (Why is that?)
• A stochastic subgradient method for distributionally robust non-convex and non-smooth learning
• Robust grouped variable selection using distributionally robust optimization
• Continuous treatment effect estimation via generative adversarial de-confounding
• Superquantiles at work: machine learning applications and efficient subgradient computation
• Policy Learning with Asymmetric Counterfactual Utilities
• Robust Validation: Confident Predictions Even When Distributions Shift
• Distributionally robust optimization. A review on theory and applications
• Towards out of distribution generalization for problems in mechanics
• A survey of learning criteria going beyond the usual risk
• Counterfactual sensitivity and robustness
• Robust and distributionally robust optimization models for linear support vector machine

Uses Software

• UCI-ml
• Caffe
• Convex.jl

This page was built for publication: Learning models with uniform performance via distributionally robust optimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q820804)