Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing

DOI10.1214/16-EJS1212MaRDI QIDQ502845zbMATHOpenAlexFDO

Authors Jelena Bradic

Publication date 11 January 2017

Published in Electronic Journal of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1507.08726

zbMATH Keywords

LassoefficiencyrobustnesssparsityLADapproximate message passingAMP

Mathematics Subject Classification ID

Nonparametric robustness (62G35) Ridge regression; shrinkage estimators (Lasso) (62J07) Central limit and other weak theorems (60F05)

Abstract: Understanding efficiency in high dimensional linear models is a longstanding problem of interest. Classical work with smaller dimensional problems dating back to Huber and Bickel has illustrated the benefits of efficient loss functions. When the number of parameters $p$ is of the same order as the sample size $n$, $papproxn$, an efficiency pattern different from the one of Huber was recently established. In this work, we consider the effects of model selection on the estimation efficiency of penalized methods. In particular, we explore whether sparsity, results in new efficiency patterns when $p>n$. In the interest of deriving the asymptotic mean squared error for regularized M-estimators, we use the powerful framework of approximate message passing. We propose a novel, robust and sparse approximate message passing algorithm (RAMP), that is adaptive to the error distribution. Our algorithm includes many non-quadratic and non-differentiable loss functions. We derive its asymptotic mean squared error and show its convergence, while allowing $p,n,soinfty$, with $n/pin(0,1)$ and $n/sin(1,infty)$. We identify new patterns of relative efficiency regarding a number of penalized $M$ estimators, when $p$ is much larger than $n$. We show that the classical information bound is no longer reachable, even for light--tailed error distributions. We show that the penalized least absolute deviation estimator dominates the penalized least square estimator, in cases of heavy--tailed distributions. We observe this pattern for all choices of the number of non-zero parameters $s$, both $sleqn$ and $sapproxn$. In non-penalized problems where $s=papproxn$, the opposite regime holds. Therefore, we discover that the presence of model selection significantly changes the efficiency patterns.

Recommendations

• High dimensional robust M-estimation: asymptotic variance via approximate message passing
• Robust error density estimation in ultrahigh dimensional sparse linear model
• Robust and sparse multinomial regression in high dimensions
• Penalised robust estimators for sparse and high-dimensional linear models
• Robustness to Unknown Error in Sparse Regularization
• Approximate Message Passing With Consistent Parameter Estimation and Applications to Sparse Learning
• Robust and sparse regression in generalized linear model by stochastic optimization
• Robust and sparse estimators for linear regression models
• Robust and sparse learning of varying coefficient models with high-dimensional features
• Robust sparse Gaussian graphical modeling

Cited in

(7)

• Automatic bias correction for testing in high‐dimensional linear models
• Overcoming the limitations of phase transition by higher order analysis of regularization techniques
• A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
• High dimensional robust M-estimation: asymptotic variance via approximate message passing
• Detangling robustness in high dimensions: composite versus model-averaged estimation
• Scale calibration for high-dimensional robust regression
• Asymptotic risk and phase transition of \(l_1\)-penalized robust estimator

This page was built for publication: Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q502845)