Robust linear least squares regression

DOI10.1214/11-AOS918MaRDI QIDQ661182zbMATH OpenOpenAlexWikidataFDO

Authors Jean-Yves Audibert, Olivier Catoni

Publication date 21 February 2012

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1010.0074

zbMATH Keywords

generalization errorGibbs posterior distributionsPAC-Bayesian theoremsrandomized estimatorsresistant estimatorsrisk boundsshrinkagestatistical learning theory

Mathematics Subject Classification ID

Minimax procedures in statistical decision theory (62C20) Parametric inference under constraints (62F30) Robustness and adaptive procedures (parametric inference) (62F35) Linear regression; mixed models (62J05) Ridge regression; shrinkage estimators (Lasso) (62J07)

Abstract: We consider the problem of robustly predicting as well as the best linear combination of $d$ given functions in least squares regression, and variants of this problem including constraints on the parameters of the linear combination. For the ridge estimator and the ordinary least squares estimator, and their variants, we provide new risk bounds of order $d/n$ without logarithmic factor unlike some standard results, where $n$ is the size of the training data. We also provide a new estimator with better deviations in the presence of heavy-tailed noise. It is based on truncating differences of losses in a min--max framework and satisfies a $d/n$ risk bound both in expectation and in deviations. The key common surprising factor of these results is the absence of exponential moment condition on the output distribution while achieving exponential deviations. All risk bounds are obtained through a PAC-Bayesian analysis on truncated differences of losses. Experimental results strongly back up our truncated min--max estimator.

Recommendations

• Distribution-free robust linear regression
• Loss minimization and parameter estimation with heavy tails
• On robust regression with high-dimensional predictors
• Iteratively reweighted _1-penalized robust regression
• Excess risk bounds in robust empirical risk minimization

Cites work

• A distribution-free theory of nonparametric regression
• Aggregating regression procedures to improve performance
• Challenging the empirical mean and empirical variance: a deviation study
• High breakdown-point and high efficiency robust estimates for regression
• scientific article; zbMATH DE number 3905646 (Why is no real title available?)
• scientific article; zbMATH DE number 1522808 (Why is no real title available?)
• Learning Theory and Kernel Machines
• Minimum contrast estimators on sieves: Exponential bounds and rates of convergence
• Model selection for regression on a fixed design
• Piecewise Polynomial Estimation of a Regression Function
• Robust linear least squares regression

Cited in

(61)

• Geometric median and robust estimation in Banach spaces
• Robust linear regression with broad distributions of errors
• Simpler PAC-Bayesian bounds for hostile data
• Robust regression using biased objectives
• Challenging the empirical mean and empirical variance: a deviation study
• PAC-Bayesian estimation and prediction in sparse additive models
• Rho-estimators revisited: general theory and applications
• Estimation from nonlinear observations via convex programming with application to bilinear regression
• Learning from MOM's principles: Le Cam's approach
• Finite impulse response models: a non-asymptotic analysis of the least squares estimator
• A MOM-based ensemble method for robustness, subsampling and hyperparameter tuning
• Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
• A minimax framework for quantifying risk-fairness trade-off in regression
• Suboptimality of constrained least squares and improvements via non-linear predictors
• Distribution-free robust linear regression
• On least squares estimation under heteroscedastic and heavy-tailed errors
• Aggregated hold out for sparse linear regression with a robust loss function
• Robust statistical learning with Lipschitz and convex loss functions
• Confidence regions and minimax rates in outlier-robust estimation on the probability simplex
• Robust machine learning by median-of-means: theory and practice
• Robust inference via multiplier bootstrap
• Robust classification via MOM minimization
• Finite sample performance of linear least squares estimation
• Algorithms of robust stochastic optimization based on mirror descent method
• Convergence rates of least squares regression estimators with heavy-tailed errors
• Mean estimation and regression under heavy-tailed distributions: A survey
• Regularization, sparse recovery, and median-of-means tournaments
• Empirical risk minimization is optimal for the convex aggregation problem
• On some recent advances on high dimensional Bayesian statistics
• Performance of empirical risk minimization in linear aggregation
• On the properties of variational approximations of Gibbs posteriors
• The lower tail of random quadratic forms with applications to ordinary least squares
• Random design analysis of ridge regression
• Robust lagfactors
• Robust Linear Regression via \ell₀ Regularization
• scientific article; zbMATH DE number 7370566 (Why is no real title available?)
• A New Principle for Tuning-Free Huber Regression
• A User-Friendly Computational Framework for Robust Structured Regression with the L₂ Criterion
• A new method for estimation and model selection: \(\rho\)-estimation
• Fast and approximate exhaustive variable selection for generalised linear models with APES
• Robust high dimensional learning for Lipschitz and convex losses
• First-order convergence theory for weakly-convex-weakly-concave min-max problems
• User-friendly Introduction to PAC-Bayes Bounds
• Dimension-free bounds for sums of independent matrices and simple tensors via the variational principle
• Robust subgaussian estimation with VC-dimension
• ARFIS: an adaptive robust model for regression with heavy-tailed distribution
• Robust linear least squares regression
• Least squares regression under weak moment conditions
• Covariance estimation: optimal dimension-free guarantees for adversarial corruption and heavy tails
• Performance of empirical risk minimization for linear regression with dependent data
• Nonsmooth nonconvex-nonconcave minimax optimization: primal-dual balancing and iteration complexity analysis
• Trimmed sample means for robust uniform mean estimation and regression
• Distributionally robust optimization and robust statistics
• Statistically optimal robust mean and covariance estimation for anisotropic gaussians
• On deviation probabilities in non-parametric regression with heavy-tailed noise
• Robust Regression with Covariate Filtering: Heavy Tails and Adversarial Contamination
• Generalized median of means principle for Bayesian inference
• Noisy recovery from random linear observations: sharp minimax rates under elliptical constraints
• Statistical analysis of the moving least-squares method with unbounded sampling
• Comparison theorems for the minimum eigenvalue of a random positive-semidefinite matrix
• Empirical risk minimization for heavy-tailed losses

This page was built for publication: Robust linear least squares regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q661182)