High dimensional robust M-estimation: asymptotic variance via approximate message passing
From MaRDI portal
(Redirected from Publication:343797)
approximate message passingapproximate message passing (AMP)asymptotic varianceFisher informationGaussian noiserobust M-estimationrobust regression estimators
Random matrices (probabilistic aspects) (60B20) Asymptotic properties of parametric estimators (62F12) Robustness and adaptive procedures (parametric inference) (62F35) Estimation in multivariate analysis (62H12) Linear regression; mixed models (62J05) Ridge regression; shrinkage estimators (Lasso) (62J07) Analysis of variance and covariance (ANOVA) (62J10)
Abstract: In a recent article (Proc. Natl. Acad. Sci., 110(36), 14557-14562), El Karoui et al. study the distribution of robust regression estimators in the regime in which the number of parameters p is of the same order as the number of samples n. Using numerical simulations and `highly plausible' heuristic arguments, they unveil a striking new phenomenon. Namely, the regression coefficients contain an extra Gaussian noise component that is not explained by classical concepts such as the Fisher information matrix. We show here that that this phenomenon can be characterized rigorously techniques that were developed by the authors to analyze the Lasso estimator under high-dimensional asymptotics. We introduce an approximate message passing (AMP) algorithm to compute M-estimators and deploy state evolution to evaluate the operating characteristics of AMP and so also M-estimates. Our analysis clarifies that the `extra Gaussian noise' encountered in this problem is fundamentally similar to phenomena already studied for regularized least squares in the setting n<p.
Recommendations
- On robust regression with high-dimensional predictors
- Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing
- Penalised robust estimators for sparse and high-dimensional linear models
- Statistical consistency and asymptotic normality for high-dimensional robust \(M\)-estimators
- Asymptotic risk and phase transition of \(l_1\)-penalized robust estimator
Cites work
- An introduction to random matrices
- An iterative construction of solutions of the TAP equations for the Sherrington-Kirkpatrick model
- Applications of the Lindeberg Principle in Communications and Statistical Learning
- Construction of pure states in mean field models for spin glasses
- Finding hidden cliques of size \(\sqrt{N/e}\) in nearly linear time
- scientific article; zbMATH DE number 1273988 (Why is no real title available?)
- scientific article; zbMATH DE number 845714 (Why is no real title available?)
- Information, Physics, and Computation
- Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing
- On robust regression with high-dimensional predictors
- One-Step Huber Estimates in the Linear Model
- Rigorous solution of the Gardner problem
- Robust Estimation of a Location Parameter
- Robust regression: Asymptotics, conjectures and Monte Carlo
- Robust Statistics
- State evolution for general approximate message passing algorithms, with applications to spatial coupling
- The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
- The LASSO Risk for Gaussian Matrices
- The Noise-Sensitivity Phase Transition in Compressed Sensing
- Universality in polytope phase transitions and message passing algorithms
Cited in
(77)- Asymptotics for high dimensional regression \(M\)-estimates: fixed design results
- Overcoming the limitations of phase transition by higher order analysis of regularization techniques
- Distributed linear regression by averaging
- High-dimensional linear models: a random matrix perspective
- The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
- High-dimensional robust approximated M-estimators for mean regression with asymmetric data
- Precise statistical analysis of classification accuracies for adversarial training
- Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
- Asymptotic normality of robust M-estimators with convex penalty
- Fundamental barriers to high-dimensional regression with convex penalties
- Approximate message passing algorithms for rotationally invariant matrices
- Activation function design for deep networks: linearity and effective initialisation
- The asymptotic distribution of the MLE in high-dimensional logistic models: arbitrary covariance
- A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
- Detangling robustness in high dimensions: composite versus model-averaged estimation
- Confidence regions and minimax rates in outlier-robust estimation on the probability simplex
- Which bridge estimator is the best for variable selection?
- Asymptotic risk and phase transition of \(l_1\)-penalized robust estimator
- Finite-sample analysis of \(M\)-estimators using self-concordance
- The likelihood ratio test in high-dimensional logistic regression is asymptotically a rescaled Chi-square
- Multidimensional linear functional estimation in sparse Gaussian models and robust estimation of the mean
- Fundamental limits of weak recovery with applications to phase retrieval
- SLOPE is adaptive to unknown sparsity and asymptotically minimax
- On robust regression with high-dimensional predictors
- Statistical mechanics of the inverse Ising problem and the optimal objective function
- Concentration inequalities for statistical inference
- Can we trust the bootstrap in high-dimensions? The case of linear models
- Statistics of robust optimization: a generalized empirical likelihood approach
- Approximate survey propagation for statistical inference
- Robustness in sparse high-dimensional linear models: relative efficiency and robust approximate message passing
- Learning curves of generic features maps for realistic datasets with a teacher-student model*
- The scaling limit of high-dimensional online independent component analysis
- Entropy and mutual information in models of deep neural networks*
- Generalized approximate survey propagation for high-dimensional estimation *
- A Unifying Tutorial on Approximate Message Passing
- Automatic bias correction for testing in high‐dimensional linear models
- Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares
- Debiasing convex regularized estimators and interval estimation in linear models
- Approximate message passing for sparse matrices with application to the equilibria of large ecological Lotka-Volterra systems
- A Friendly Tutorial on Mean-Field Spin Glass Techniques for Non-Physicists
- Universality of approximate message passing with semirandom matrices
- Noisy linear inverse problems under convex constraints: exact risk asymptotics in high dimensions
- Universality of regularized regression estimators in high dimensions
- Adaptive and robust multi-task learning
- The Lasso with general Gaussian designs with applications to hypothesis testing
- A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression
- Robust and tuning-free sparse linear regression via square-root slope
- Robust estimation of covariance matrices: adversarial contamination and beyond
- Fluctuations, bias, variance and ensemble of learners: exact asymptotics for convex losses in high-dimension
- Universality of approximate message passing algorithms and tensor networks
- Equilibria of large random Lotka-Volterra systems with vanishing species: a mathematical approach
- An adaptively resized parametric bootstrap for inference in high-dimensional generalized linear models
- On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators
- Optimal convex M-estimation via score matching
- Kronecker-product random matrices and a matrix least squares problem
- Capacity lower bound for the Ising perceptron
- Learning from similar linear representations: adaptivity, minimaxity, and robustness
- Distributionally robust optimization and robust statistics
- Spectral estimators for structured generalized linear models via approximate message passing
- Correlation adjusted debiased Lasso: debiasing the Lasso with inaccurate covariate model
- Entrywise dynamics and universality of general first order methods
- A leave-one-out approach to approximate message passing
- High-dimensional learning of narrow neural networks
- The generalization error of max-margin linear classifiers: benign overfitting and high dimensional asymptotics in the overparametrized regime
- A new central limit theorem for the augmented IPW estimator: variance inflation, cross-fit covariance and beyond
- Observable adjustments in single-index models for regularized M-estimators with bounded p/n
- Error estimation and adaptive tuning for unregularized robust M-estimator
- Equivalence of state equations from different methods in high-dimensional regression
- Dimension free ridge regression
- Spectrum-aware debiasing: a modern inference framework with applications to principal components regression
- Precise asymptotics of bagging regularized M-estimators
- High-dimensional robust regression under heavy-tailed data: asymptotics and universality
- Analysis of high-dimensional Gaussian labeled-unlabeled mixture model via message-passing algorithm
- A phase transition between positional and semantic learning in a solvable model of dot-product attention
- Universality of estimators for high-dimensional linear models with block dependency
- Diaconis-Ylvisaker prior penalized likelihood for p/n(0, 1) logistic regression
- The main contributions of robust statistics to statistical science and a new challenge
This page was built for publication: High dimensional robust M-estimation: asymptotic variance via approximate message passing
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q343797)
