Characterization of the equivalence of robustification and regularization in linear and matrix regression

DOI10.1016/J.EJOR.2017.03.051zbMATH Open1403.62040arXiv1411.6160OpenAlexW2963772730WikidataQ89226031 ScholiaQ89226031MaRDI QIDQ723995FDOQ723995

Authors: Martin S. Copenhaver, Dimitris Bertsimas Error creating thumbnail:

Publication date: 25 July 2018

Published in: European Journal of Operational Research (Search for Journal in Brave)

Abstract: The notion of developing statistical methods in machine learning which are robust to adversarial perturbations in the underlying data has been the subject of increasing interest in recent years. A common feature of this work is that the adversarial robustification often corresponds exactly to regularization methods which appear as a loss function plus a penalty. In this paper we deepen and extend the understanding of the connection between robustification and regularization (as achieved by penalization) in regression problems. Specifically, (a) in the context of linear regression, we characterize precisely under which conditions on the model of uncertainty used and on the loss function penalties robustification and regularization are equivalent, and (b) we extend the characterization of robustification and regularization to matrix regression problems (matrix completion and Principal Component Analysis).

Full work available at URL: https://arxiv.org/abs/1411.6160

Recommendations

• The generalized equivalence of regularization and min-max robustification in linear mixed models
• Robustness and regularization of support vector machines
• Robust regression via mutivariate regression depth
• Robust regression using biased objectives
• Robust classification and regression using support vector machines

zbMATH Keywords

convex programmingrobust optimizationpenalty methodsadversarial learningstatistical regression

Mathematics Subject Classification ID

Factor analysis and principal components; correspondence analysis (62H25) Learning and adaptive systems in artificial intelligence (68T05) Convex programming (90C25) Estimation in multivariate analysis (62H12) Ridge regression; shrinkage estimators (Lasso) (62J07) Robustness and adaptive procedures (parametric inference) (62F35)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Robust regression: Asymptotics, conjectures and Monte Carlo
• A survey of robust statistics
• Title not available (Why is that?)
• Regularization and Variable Selection Via the Elastic Net
• Measurement Error in Nonlinear Models
• Convex analysis and monotone operator theory in Hilbert spaces
• Adaptive robust variable selection
• On combining machine learning with decision making
• Robust principal component analysis?
• Least Median of Squares Regression
• Title not available (Why is that?)
• High breakdown estimators for principal components: the projection-pursuit approach revis\-ited
• Principal Components Analysis Based on Multivariate MM Estimators With Fast and Robust Bootstrap
• Robust Statistics
• High-breakdown robust multivariate methods
• The Influence Curve and Its Role in Robust Estimation
• Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization
• An Analysis of the Total Least Squares Problem
• Overview of total least-squares methods
• Theory and applications of robust optimization
• Robust optimization
• Robust Solutions to Least-Squares Problems with Uncertain Data
• Data-driven robust optimization
• Consistency of the structured total least squares estimator in a multivariate errors-in-variables model
• Elastic-net regularization in learning theory
• Penalized Composite Quasi-Likelihood for Ultrahigh Dimensional Variable Selection
• Robust Regression and Lasso
• Lipschitz behavior of the robust regularization
• Oracle-based robust optimization via online learning

Cited In (35)

• Robust linear algebra
• Certifiably optimal sparse principal component analysis
• Rejoinder: ``Sparse regression: scalable algorithms and empirical performance
• An Analytical and Geometric Perspective on Adversarial Robustness
• Regularization via mass transportation
• Sparse classification: a scalable discrete optimization perspective
• Sparse regression for large data sets with outliers
• A look at robustness and stability of \(\ell_1\)-versus \(\ell_0\)-regularization: discussion of papers by Bertsimas et al. and Hastie et al.
• Sparse regression over clusters: SparClur
• An Alternating Method for Cardinality-Constrained Optimization: A Computational Study for the Best Subset Selection and Sparse Portfolio Problems
• Certifiably optimal sparse inverse covariance estimation
• Optimization problems for machine learning: a survey
• Robust prediction of domain compositions from uncertain data using isometric logratio transformations in a penalized multivariate Fay–Herriot model
• The backbone method for ultra-high dimensional sparse machine learning
• Calibration of distributionally robust empirical optimization models
• OR forum: An algorithmic approach to linear regression
• Detecting racial bias in jury selection
• Robust Linear Regression via $\ell_0$ Regularization
• Robust grouped variable selection using distributionally robust optimization
• Robust newsvendor problem with autoregressive demand
• The generalized equivalence of regularization and min-max robustification in linear mixed models
• Title not available (Why is that?)
• Robust \(\mathcal S\)-regularity of matrix pencils applied to the analysis of descriptor models
• A robust learning approach for regression models based on distributionally robust optimization
• Sparse hierarchical regression with polynomials
• Diametrical risk minimization: theory and computations
• Scalable holistic linear regression
• Ambulance emergency response optimization in developing countries
• Near-optimal nonlinear regression trees
• Distributionally robust optimization. A review on theory and applications
• Learning sparse nonlinear dynamics via mixed-integer optimization
• Adversarial classification via distributional robustness with Wasserstein ambiguity
• RSG: Beating Subgradient Method without Smoothness and Strong Convexity
• Robust Wasserstein profile inference and applications to machine learning
• An elastic net penalized small area model combining unit- and area-level data for regional hypertension prevalence estimation

Uses Software

• ElemStatLearn
• ROBPCA

This page was built for publication: Characterization of the equivalence of robustification and regularization in linear and matrix regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q723995)