A statistical learning assessment of Huber regression

DOI10.1016/J.JAT.2021.105660MaRDI QIDQ2054280zbMATH OpenOpenAlexFDO

Authors Qiang Wu, Yun-Long Feng

Publication date 1 December 2021

Published in Journal of Approximation Theory (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2009.12755

zbMATH Keywords

convergence ratesHuber regressionlearning theoryrobustnessweak moment condition

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Robustness and adaptive procedures (parametric inference) (62F35) Linear regression; mixed models (62J05) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: As one of the triumphs and milestones of robust statistics, Huber regression plays an important role in robust inference and estimation. It has also been finding a great variety of applications in machine learning. In a parametric setup, it has been extensively studied. However, in the statistical learning context where a function is typically learned in a nonparametric way, there is still a lack of theoretical understanding of how Huber regression estimators learn the conditional mean function and why it works in the absence of light-tailed noise assumptions. To address these fundamental questions, we conduct an assessment of Huber regression from a statistical learning viewpoint. First, we show that the usual risk consistency property of Huber regression estimators, which is usually pursued in machine learning, cannot guarantee their learnability in mean regression. Second, we argue that Huber regression should be implemented in an adaptive way to perform mean regression, implying that one needs to tune the scale parameter in accordance with the sample size and the moment condition of the noise. Third, with an adaptive choice of the scale parameter, we demonstrate that Huber regression estimators can be asymptotic mean regression calibrated under $(1+epsilon)$-moment conditions ($epsilon>0$). Last but not least, under the same moment conditions, we establish almost sure convergence rates for Huber regression estimators. Note that the $(1+epsilon)$-moment conditions accommodate the special case where the response variable possesses infinite variance and so the established convergence rates justify the robustness feature of Huber regression estimators. In the above senses, the present study provides a systematic statistical learning assessment of Huber regression estimators and justifies their merits in terms of robustness from a theoretical viewpoint.

Recommendations

• Adaptive Huber Regression
• Scale calibration for high-dimensional robust regression
• Robust regression via mutivariate regression depth
• Sharp non-asymptotic performance bounds for _1 and Huber robust regression estimators
• Iteratively reweighted _1-penalized robust regression

Cites work

• A general Bahadur representation of M-estimators and its application to linear regression with nonstochastic designs
• Adaptive Huber Regression
• Are Loss Functions All the Same?
• Asymptotic behavior of M-estimators for the linear model
• Asymptotic behavior of M-estimators of p regression parameters when p^ 2/n is large. I. Consistency
• Balancing principle in supervised learning for a general regularization scheme
• Consistency and robustness of kernel-based regression in convex risk minimization
• Cross-validation based adaptation for regularization operators in learning theory
• Empirical minimization
• Fast rates for support vector machines using Gaussian kernels
• Fast rates in statistical and online learning
• scientific article; zbMATH DE number 1332320 (Why is no real title available?)
• Learning Theory
• On regularization algorithms in learning theory
• Regularized Nyström subsampling in regression and ranking problems under general smoothness assumptions
• Robust Estimation of a Location Parameter
• Robust regression: Asymptotics, conjectures and Monte Carlo
• Robust statistical learning with Lipschitz and convex loss functions
• Robust Statistics
• Robust Statistics
• Statistical consistency and asymptotic normality for high-dimensional robust \(M\)-estimators
• Support Vector Machines
• The elements of statistical learning. Data mining, inference, and prediction

Cited in

(22)

• Robust pairwise learning with Huber loss
• Functional linear regression with Huber loss
• Safe feature screening rules for the regularized Huber regression
• Nonasymptotic analysis of robust regression with modified Huber's loss
• Adaptive Huber Regression
• A New Principle for Tuning-Free Huber Regression
• Learning theory of minimum error entropy under weak moment conditions
• scientific article; zbMATH DE number 7730941 (Why is no real title available?)
• Least squares regression under weak moment conditions
• Convergence rates of regularized Huber regression under weak moment conditions
• Robust learning of the modified Huber loss
• Robust transfer learning for high-dimensional regression with linear constraints
• Tikhonov regularization for Gaussian empirical gain maximization in RKHS is consistent
• Understanding generalization of functional linear regression from non-i.i.d. sample viewpoint
• Kernel mode-based varying coefficient models with nonstationary regressors
• Robust learning of Huber loss under exponentially strongly mixing sequence
• Learning theory of regularized Huber regression
• Improved Huber regression in distributed systems
• ℓ ₁ -regularization learning based on Huber regression
• Classification with shallow neural networks
• Learning under \((1 + \epsilon)\)-moment conditions
• Sharp non-asymptotic performance bounds for _1 and Huber robust regression estimators

Describes a project that uses

Uses Software

• robustbase

This page was built for publication: A statistical learning assessment of Huber regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2054280)