Universality of regularized regression estimators in high dimensions

DOI10.1214/23-AOS2309MaRDI QIDQ6183759zbMATH OpenFDO

Authors Qiyang Han, Yandi Shen

Publication date 4 January 2024

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2206.07936, https://projecteuclid.org/journals/annals-of-statistics/volume-51/issue-4/Universality-of-regularized-regression-estimators-in-high-dimensions/10.1214/23-AOS2309.full

zbMATH Keywords

Lassoridge regressionrandom matrix theoryrobust regressionhigh-dimensional asymptoticsuniversalityGaussian comparison inequalitiesLindeberg's principle

Mathematics Subject Classification ID

Random matrices (probabilistic aspects) (60B20) Approximations to statistical distributions (nonasymptotic) (62E17) Functional limit theorems; invariance principles (60F17)

Abstract: The Convex Gaussian Min-Max Theorem (CGMT) has emerged as a prominent theoretical tool for analyzing the precise stochastic behavior of various statistical estimators in the so-called high dimensional proportional regime, where the sample size and the signal dimension are of the same order. However, a well recognized limitation of the existing CGMT machinery rests in its stringent requirement on the exact Gaussianity of the design matrix, therefore rendering the obtained precise high dimensional asymptotics largely a specific Gaussian theory in various important statistical models. This paper provides a structural universality framework for a broad class of regularized regression estimators that is particularly compatible with the CGMT machinery. In particular, we show that with a good enough $el{l}_{i}nfty$ bound for the regression estimator $hat{mu}_A$, any `structural property' that can be detected via the CGMT for $hat{mu}_G$ (under a standard Gaussian design $G$) also holds for $hat{mu}_A$ under a general design $A$ with independent entries. As a proof of concept, we demonstrate our new universality framework in three key examples of regularized regression estimators: the Ridge, Lasso and regularized robust regression estimators, where new universality properties of risk asymptotics and/or distributions of regression estimators and other related quantities are proved. As a major statistical implication of the Lasso universality results, we validate inference procedures using the degrees-of-freedom adjusted debiased Lasso under general design and error distributions. We also provide a counterexample, showing that universality properties for regularized regression estimators do not extend to general isotropic designs.

Recommendations

• A unified framework for high-dimensional analysis of \(M\)-estimators with decomposable regularizers
• Debiasing the Lasso: optimal sample size for Gaussian designs
• Fundamental barriers to high-dimensional regression with convex penalties
• The Lasso with general Gaussian designs with applications to hypothesis testing
• Asymptotics for high dimensional regression \(M\)-estimates: fixed design results

Cites work

• scientific article; zbMATH DE number 4061904 (Why is no real title available?)
• scientific article; zbMATH DE number 1273988 (Why is no real title available?)
• scientific article; zbMATH DE number 845714 (Why is no real title available?)
• A generalization of the Lindeberg principle
• A model of double descent for high-dimensional binary linear classification
• A modern maximum-likelihood theory for high-dimensional logistic regression
• A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
• Applications of the Lindeberg Principle in Communications and Statistical Learning
• Approximate message passing algorithms for rotationally invariant matrices
• Central limit theorem and bootstrap approximation in high dimensions: near \(1/\sqrt{n}\) rates via implicit smoothing
• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Confidence intervals for low dimensional parameters in high dimensional linear models
• De-biasing the Lasso with degrees-of-freedom adjustment
• Debiasing convex regularized estimators and interval estimation in linear models
• Debiasing the Lasso: optimal sample size for Gaussian designs
• Does SLOPE outperform bridge regression?
• Estimation of the mean of a multivariate normal distribution
• Fundamental barriers to high-dimensional regression with convex penalties
• Fundamental limits of symmetric low-rank matrix estimation
• Generalisation error in learning with random features and the hidden manifold model*
• High dimensional robust M-estimation: asymptotic variance via approximate message passing
• High-dimensional asymptotics of prediction: ridge regression and classification
• High-dimensional central limit theorems by Stein's method
• Hypothesis Testing in High-Dimensional Regression Under the Gaussian Random Design Model: Asymptotic Theory
• Learning curves of generic features maps for realistic datasets with a teacher-student model*
• Mean Field Models for Spin Glasses
• Mean field asymptotics in high-dimensional statistics: from exact results to efficient algorithms
• Mean field models for spin glasses. Volume I: Basic examples.
• Nearly optimal central limit theorem and bootstrap approximations in high dimensions
• On asymptotically optimal confidence regions and tests for high-dimensional models
• On robust regression with high-dimensional predictors
• On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators
• Optimal errors and phase transitions in high-dimensional generalized linear models
• Precise Error Analysis of Regularized <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math> </inline-formula>-Estimators in High Dimensions
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Ridge regression and asymptotic minimax estimation over spheres of growing dimension
• Robust Estimation of a Location Parameter
• Robust regression: Asymptotics, conjectures and Monte Carlo
• Second-order Stein: SURE for SURE and other applications in high-dimensional inference
• Surprises in high-dimensional ridgeless least squares interpolation
• The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• The LASSO Risk for Gaussian Matrices
• The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
• Universality in polytope phase transitions and message passing algorithms
• Universality laws for randomized dimension reduction, with applications
• Universality of approximate message passing algorithms
• Weak convergence and empirical processes. With applications to statistics

Cited in

(4)

• Universal Regularization Methods: Varying the Power, the Smoothness and the Accuracy
• Regularization after retention in ultrahigh dimensional linear regression models
• Regularized parameter estimation of high dimensional distribution
• Approximate message passing with rigorous guarantees for pooled data and quantitative group testing

This page was built for publication: Universality of regularized regression estimators in high dimensions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6183759)