High dimensional robust M-estimation: asymptotic variance via approximate message passing

DOI10.1007/S00440-015-0675-ZzbMATH Open1357.62220arXiv1310.7320OpenAlexW2108394050WikidataQ59478906 ScholiaQ59478906MaRDI QIDQ343797FDOQ343797

Authors: Andrea Montanari, David Donoho 

Publication date: 29 November 2016

Published in: Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete (Search for Journal in Brave)

Abstract: In a recent article (Proc. Natl. Acad. Sci., 110(36), 14557-14562), El Karoui et al. study the distribution of robust regression estimators in the regime in which the number of parameters p is of the same order as the number of samples n. Using numerical simulations and `highly plausible' heuristic arguments, they unveil a striking new phenomenon. Namely, the regression coefficients contain an extra Gaussian noise component that is not explained by classical concepts such as the Fisher information matrix. We show here that that this phenomenon can be characterized rigorously techniques that were developed by the authors to analyze the Lasso estimator under high-dimensional asymptotics. We introduce an approximate message passing (AMP) algorithm to compute M-estimators and deploy state evolution to evaluate the operating characteristics of AMP and so also M-estimates. Our analysis clarifies that the `extra Gaussian noise' encountered in this problem is fundamentally similar to phenomena already studied for regularized least squares in the setting n<p.

Full work available at URL: https://arxiv.org/abs/1310.7320

Recommendations

• On robust regression with high-dimensional predictors
• Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing
• Penalised robust estimators for sparse and high-dimensional linear models
• Statistical consistency and asymptotic normality for high-dimensional robust \(M\)-estimators
• Asymptotic risk and phase transition of \(l_1\)-penalized robust estimator

zbMATH Keywords

asymptotic varianceFisher informationGaussian noiseapproximate message passingapproximate message passing (AMP)robust M-estimationrobust regression estimators

Mathematics Subject Classification ID

Asymptotic properties of parametric estimators (62F12) Linear regression; mixed models (62J05) Estimation in multivariate analysis (62H12) Ridge regression; shrinkage estimators (Lasso) (62J07) Analysis of variance and covariance (ANOVA) (62J10) Random matrices (probabilistic aspects) (60B20) Robustness and adaptive procedures (parametric inference) (62F35)

Cites Work

• Title not available (Why is that?)
• Robust regression: Asymptotics, conjectures and Monte Carlo
• Robust Estimation of a Location Parameter
• Robust Statistics
• Finding hidden cliques of size \(\sqrt{N/e}\) in nearly linear time
• Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing
• An introduction to random matrices
• Title not available (Why is that?)
• One-Step Huber Estimates in the Linear Model
• The LASSO Risk for Gaussian Matrices
• Information, Physics, and Computation
• Universality in polytope phase transitions and message passing algorithms
• The Noise-Sensitivity Phase Transition in Compressed Sensing
• The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
• Rigorous solution of the Gardner problem
• On robust regression with high-dimensional predictors
• An iterative construction of solutions of the TAP equations for the Sherrington-Kirkpatrick model
• State evolution for general approximate message passing algorithms, with applications to spatial coupling
• Applications of the Lindeberg Principle in Communications and Statistical Learning
• Construction of pure states in mean field models for spin glasses

Cited In (53)

• Universality of approximate message passing algorithms and tensor networks
• Equilibria of large random Lotka-Volterra systems with vanishing species: a mathematical approach
• An adaptively resized parametric bootstrap for inference in high-dimensional generalized linear models
• The main contributions of robust statistics to statistical science and a new challenge
• Asymptotic normality of robust \(M\)-estimators with convex penalty
• Overcoming the limitations of phase transition by higher order analysis of regularization techniques
• Finite-sample analysis of \(M\)-estimators using self-concordance
• Asymptotic risk and phase transition of \(l_1\)-penalized robust estimator
• A Unifying Tutorial on Approximate Message Passing
• Learning curves of generic features maps for realistic datasets with a teacher-student model*
• Concentration Inequalities for Statistical Inference
• Approximate message passing algorithms for rotationally invariant matrices
• Robust and tuning-free sparse linear regression via square-root slope
• The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
• Asymptotics for high dimensional regression \(M\)-estimates: fixed design results
• Fundamental limits of weak recovery with applications to phase retrieval
• SLOPE is adaptive to unknown sparsity and asymptotically minimax
• Confidence regions and minimax rates in outlier-robust estimation on the probability simplex
• Robust estimation of covariance matrices: adversarial contamination and beyond
• Activation function design for deep networks: linearity and effective initialisation
• The asymptotic distribution of the MLE in high-dimensional logistic models: arbitrary covariance
• Fluctuations, bias, variance and ensemble of learners: exact asymptotics for convex losses in high-dimension
• Statistical mechanics of the inverse Ising problem and the optimal objective function
• Universality of regularized regression estimators in high dimensions
• Noisy linear inverse problems under convex constraints: exact risk asymptotics in high dimensions
• Automatic bias correction for testing in high‐dimensional linear models
• Which bridge estimator is the best for variable selection?
• Adaptive and robust multi-task learning
• Precise statistical analysis of classification accuracies for adversarial training
• Distributed linear regression by averaging
• The scaling limit of high-dimensional online independent component analysis
• High-dimensional linear models: a random matrix perspective
• A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
• The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
• Statistics of Robust Optimization: A Generalized Empirical Likelihood Approach
• Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares
• Multidimensional linear functional estimation in sparse Gaussian models and robust estimation of the mean
• Can we trust the bootstrap in high-dimension?
• Entropy and mutual information in models of deep neural networks*
• Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing
• Debiasing convex regularized estimators and interval estimation in linear models
• Detangling robustness in high dimensions: composite versus model-averaged estimation
• Approximate survey propagation for statistical inference
• High-dimensional robust approximated \(M\)-estimators for mean regression with asymmetric data
• Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
• Approximate message passing for sparse matrices with application to the equilibria of large ecological Lotka-Volterra systems
• Universality of approximate message passing with semirandom matrices
• The Lasso with general Gaussian designs with applications to hypothesis testing
• Fundamental barriers to high-dimensional regression with convex penalties
• Generalized approximate survey propagation for high-dimensional estimation ^*
• On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators
• A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
• A Friendly Tutorial on Mean-Field Spin Glass Techniques for Non-Physicists

This page was built for publication: High dimensional robust M-estimation: asymptotic variance via approximate message passing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q343797)