Statistical consistency and asymptotic normality for high-dimensional robust M-estimators

DOI10.1214/16-AOS1471zbMATH Open1371.62023arXiv1501.00312OpenAlexW2963927498MaRDI QIDQ2012209FDOQ2012209

Po-Ling Loh

Publication date: 28 July 2017

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: We study theoretical properties of regularized robust M-estimators, applicable when data are drawn from a sparse high-dimensional linear model and contaminated by heavy-tailed distributions and/or outliers in the additive errors and covariates. We first establish a form of local statistical consistency for the penalized regression estimators under fairly mild conditions on the error distribution: When the derivative of the loss function is bounded and satisfies a local restricted curvature condition, all stationary points within a constant radius of the true regression vector converge at the minimax rate enjoyed by the Lasso with sub-Gaussian errors. When an appropriate nonconvex regularizer is used in place of an l_1-penalty, we show that such stationary points are in fact unique and equal to the local oracle solution with the correct support---hence, results on asymptotic normality in the low-dimensional case carry over immediately to the high-dimensional setting. This has important implications for the efficiency of regularized nonconvex M-estimators when the errors are heavy-tailed. Our analysis of the local curvature of the loss function also has useful consequences for optimization when the robust regression function and/or regularizer is nonconvex and the objective function possesses stationary points outside the local region. We show that as long as a composite gradient descent algorithm is initialized within a constant radius of the true regression vector, successive iterates will converge at a linear rate to a stationary point within the local region. Furthermore, the global optimum of a convex regularized robust regression function may be used to obtain a suitable initialization. The result is a novel two-step procedure that uses a convex M-estimator to achieve consistency and a nonconvex M-estimator to increase efficiency.

Full work available at URL: https://arxiv.org/abs/1501.00312

Recommendations

• Nonconcave penalized M-estimation with a diverging number of parameters
• Iteratively reweighted \(\ell_1\)-penalized robust regression
• Penalised robust estimators for sparse and high-dimensional linear models
• Regularized \(M\)-estimators with nonconvexity: statistical and algorithmic theory for local optima
• On robust regression with high-dimensional predictors

zbMATH Keywords

asymptotic normalityhigh-dimensional statisticsnonconvex optimizationrobust regression\(M\)-estimatorsstatistical consistencysupport recovery

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Estimation in multivariate analysis (62H12) Robustness and adaptive procedures (parametric inference) (62F35)

Cited In (78)

• A shrinkage principle for heavy-tailed data: high-dimensional robust low-rank matrix recovery
• The main contributions of robust statistics to statistical science and a new challenge
• Variance prior forms for high-dimensional Bayesian variable selection
• Finite-sample analysis of \(M\)-estimators using self-concordance
• Simultaneous feature selection and outlier detection with optimality guarantees
• A New Principle for Tuning-Free Huber Regression
• High dimensional generalized linear models for temporal dependent data
• Batch policy learning in average reward Markov decision processes
• A unified theory of confidence regions and testing for high-dimensional estimating equations
• Gaining Outlier Resistance With Progressive Quantiles: Fast Algorithms and Theoretical Studies
• Tractable Bayesian Variable Selection: Beyond Normality
• The finite sample properties of sparse M-estimators with pseudo-observations
• Robustness and Tractability for Non-convex M-estimators
• Graphical-model based high dimensional generalized linear models
• Robust matrix estimations meet Frank-Wolfe algorithm
• Regression analysis: likelihood, error and entropy
• All-in-one robust estimator of the Gaussian mean
• I-LAMM for sparse learning: simultaneous control of algorithmic complexity and statistical error
• Penalised robust estimators for sparse and high-dimensional linear models
• A high-dimensional M-estimator framework for bi-level variable selection
• Nonbifurcating Phylogenetic Tree Inference via the Adaptive LASSO
• Rejoinder to “A Tuning-Free Robust and Efficient Approach to High-Dimensional Regression”
• Variable smoothing for weakly convex composite functions
• Title not available (Why is that?)
• Adaptive Huber regression on Markov-dependent data
• The robust nearest shrunken centroids classifier for high-dimensional heavy-tailed data
• A refined convergence analysis of \(\mathrm{pDCA}_{e}\) with applications to simultaneous sparse recovery and outlier detection
• Rate-optimal robust estimation of high-dimensional vector autoregressive models
• Iteratively reweighted \(\ell_1\)-penalized robust regression
• Asymptotic linear expansion of regularized M-estimators
• Robust inference for high‐dimensional single index models
• An outer-inner linearization method for non-convex and nondifferentiable composite regularization problems
• High-dimensional composite quantile regression: optimal statistical guarantees and fast algorithms
• Adaptive sparse group LASSO in quantile regression
• A statistical learning assessment of Huber regression
• The robust desparsified lasso and the focused information criterion for high-dimensional generalized linear models
• High-dimensional robust regression with \(L_q\)-loss functions
• Sparse regression for extreme values
• Wavelet-based robust estimation and variable selection in nonparametric additive models
• Additive Bayesian variable selection under censoring and misspecification
• Differentially private inference via noisy optimization
• Oracle Inequalities for Local and Global Empirical Risk Minimizers
• Scale calibration for high-dimensional robust regression
• Distributed adaptive Huber regression
• Identifiability and estimation of meta-elliptical copula generators
• Manifold Sampling for Optimization of Nonconvex Functions That Are Piecewise Linear Compositions of Smooth Components
• Asymptotic properties on high-dimensional multivariate regression M-estimation
• The landscape of empirical risk for nonconvex losses
• Models as approximations. I. Consequences illustrated with linear regression
• Statistical analysis of sparse approximate factor models
• Penalized wavelet estimation and robust denoising for irregular spaced data
• High-dimensional robust approximated \(M\)-estimators for mean regression with asymmetric data
• A Tuning-free Robust and Efficient Approach to High-dimensional Regression
• High-Dimensional Learning Under Approximate Sparsity with Applications to Nonsmooth Estimation and Regularized Neural Networks
• Computational and statistical analyses for robust non-convex sparse regularized regression problem
• Penalized wavelet nonparametric univariate logistic regression for irregular spaced data
• D4R: doubly robust reduced rank regression in high dimension
• Robust variable selection and estimation via adaptive elastic net S-estimators for linear regression
• Retire: robust expectile regression in high dimensions
• Robust High-Dimensional Regression with Coefficient Thresholding and Its Application to Imaging Data Analysis
• Overview of robust variable selection methods for high-dimensional linear regression model
• Asymptotic Properties of Stationary Solutions of Coupled Nonconvex Nonsmooth Empirical Risk Minimization
• Robust two-stage estimation in general spatial dynamic panel data models
• Asymptotic behaviour of penalized robust estimators in logistic regression when dimension increases
• Robust high-dimensional tuning free multiple testing
• Structure learning via unstructured kernel-based M-estimation
• How do noise tails impact on deep ReLU networks?
• Robust Signal Recovery for High-Dimensional Linear Log-Contrast Models with Compositional Covariates
• Inference for high-dimensional linear expectile regression with de-biasing method
• A novel robust estimation for high-dimensional precision matrices
• Renewable Huber estimation method for streaming datasets
• Comment: Feature Screening and Variable Selection via Iterative Ridge Regression
• Weakly-convex–concave min–max optimization: provable algorithms and applications in machine learning
• Sparse Laplacian shrinkage for nonparametric transformation survival model
• Fully polynomial-time randomized approximation schemes for global optimization of high-dimensional minimax concave penalized generalized linear models
• A Bernstein-type inequality for high dimensional linear processes with applications to robust estimation of time series regressions
• Analysis of global and local optima of regularized quantile regression in high dimensions: a subgradient approach
• M-estimators for models with a mix of discrete and continuous parameters

This page was built for publication: Statistical consistency and asymptotic normality for high-dimensional robust \(M\)-estimators

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2012209)