Universality of regularized regression estimators in high dimensions

DOI10.1214/23-AOS2309arXiv2206.07936MaRDI QIDQ6183759FDOQ6183759

Authors: Qiyang Han, Yandi Shen 

Publication date: 4 January 2024

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: The Convex Gaussian Min-Max Theorem (CGMT) has emerged as a prominent theoretical tool for analyzing the precise stochastic behavior of various statistical estimators in the so-called high dimensional proportional regime, where the sample size and the signal dimension are of the same order. However, a well recognized limitation of the existing CGMT machinery rests in its stringent requirement on the exact Gaussianity of the design matrix, therefore rendering the obtained precise high dimensional asymptotics largely a specific Gaussian theory in various important statistical models. This paper provides a structural universality framework for a broad class of regularized regression estimators that is particularly compatible with the CGMT machinery. In particular, we show that with a good enough $el{l}_{i}nfty$ bound for the regression estimator $hat{mu}_A$, any `structural property' that can be detected via the CGMT for $hat{mu}_G$ (under a standard Gaussian design $G$) also holds for $hat{mu}_A$ under a general design $A$ with independent entries. As a proof of concept, we demonstrate our new universality framework in three key examples of regularized regression estimators: the Ridge, Lasso and regularized robust regression estimators, where new universality properties of risk asymptotics and/or distributions of regression estimators and other related quantities are proved. As a major statistical implication of the Lasso universality results, we validate inference procedures using the degrees-of-freedom adjusted debiased Lasso under general design and error distributions. We also provide a counterexample, showing that universality properties for regularized regression estimators do not extend to general isotropic designs.

Full work available at URL: https://arxiv.org/abs/2206.07936

Recommendations

• A unified framework for high-dimensional analysis of \(M\)-estimators with decomposable regularizers
• Debiasing the Lasso: optimal sample size for Gaussian designs
• Fundamental barriers to high-dimensional regression with convex penalties
• The Lasso with general Gaussian designs with applications to hypothesis testing
• Asymptotics for high dimensional regression \(M\)-estimates: fixed design results

zbMATH Keywords

Lassoridge regressionrandom matrix theoryrobust regressionhigh-dimensional asymptoticsuniversalityGaussian comparison inequalitiesLindeberg's principle

Mathematics Subject Classification ID

Random matrices (probabilistic aspects) (60B20) Approximations to statistical distributions (nonasymptotic) (62E17) Functional limit theorems; invariance principles (60F17)

Cites Work

• Weak convergence and empirical processes. With applications to statistics
• Title not available (Why is that?)
• Estimation of the mean of a multivariate normal distribution
• Robust regression: Asymptotics, conjectures and Monte Carlo
• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Robust Estimation of a Location Parameter
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Confidence Intervals for Low Dimensional Parameters in High Dimensional Linear Models
• On asymptotically optimal confidence regions and tests for high-dimensional models
• Hypothesis Testing in High-Dimensional Regression Under the Gaussian Random Design Model: Asymptotic Theory
• Title not available (Why is that?)
• Mean Field Models for Spin Glasses
• Title not available (Why is that?)
• Mean field models for spin glasses. Volume I: Basic examples.
• High dimensional robust M-estimation: asymptotic variance via approximate message passing
• The LASSO Risk for Gaussian Matrices
• A generalization of the Lindeberg principle
• Universality in polytope phase transitions and message passing algorithms
• The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
• On robust regression with high-dimensional predictors
• Applications of the Lindeberg Principle in Communications and Statistical Learning
• Ridge regression and asymptotic minimax estimation over spheres of growing dimension
• Fundamental limits of symmetric low-rank matrix estimation
• Precise Error Analysis of Regularized <inline-formula> <tex-math notation="LaTeX">$M$ </tex-math> </inline-formula>-Estimators in High Dimensions
• On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators
• Debiasing the Lasso: optimal sample size for Gaussian designs
• Second-order Stein: SURE for SURE and other applications in high-dimensional inference
• Universality laws for randomized dimension reduction, with applications
• High-dimensional central limit theorems by Stein's method
• High-dimensional asymptotics of prediction: ridge regression and classification
• A modern maximum-likelihood theory for high-dimensional logistic regression
• Universality of approximate message passing algorithms
• The distribution of the Lasso: uniform control over sparse balls and adaptive parameter tuning
• Optimal errors and phase transitions in high-dimensional generalized linear models
• De-biasing the Lasso with degrees-of-freedom adjustment
• Surprises in high-dimensional ridgeless least squares interpolation
• Fundamental barriers to high-dimensional regression with convex penalties
• A precise high-dimensional asymptotic theory for boosting and minimum-\(\ell_1\)-norm interpolated classifiers
• The Generalization Error of Random Features Regression: Precise Asymptotics and the Double Descent Curve
• Does SLOPE outperform bridge regression?
• Central limit theorem and bootstrap approximation in high dimensions: near \(1/\sqrt{n}\) rates via implicit smoothing
• A model of double descent for high-dimensional binary linear classification
• Nearly optimal central limit theorem and bootstrap approximations in high dimensions
• Debiasing convex regularized estimators and interval estimation in linear models
• Approximate message passing algorithms for rotationally invariant matrices
• Generalisation error in learning with random features and the hidden manifold model*
• Mean field asymptotics in high-dimensional statistics: from exact results to efficient algorithms
• Learning curves of generic features maps for realistic datasets with a teacher-student model*

Cited In (4)

• Universal Regularization Methods: Varying the Power, the Smoothness and the Accuracy
• Regularization after retention in ultrahigh dimensional linear regression models
• Regularized parameter estimation of high dimensional distribution
• Approximate message passing with rigorous guarantees for pooled data and quantitative group testing

This page was built for publication: Universality of regularized regression estimators in high dimensions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6183759)