Ridge regression and asymptotic minimax estimation over spheres of growing dimension

DOI10.3150/14-BEJ609MaRDI QIDQ5963493zbMATH OpenOpenAlexFDO

Authors Lee H. Dicker

Publication date 22 February 2016

Published in Bernoulli (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1601.03900, https://projecteuclid.org/euclid.bj/1443620842

zbMATH Keywords

random matrix theoryadaptive estimationequivarianceMarčenko-Pastur distribution

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Linear regression; mixed models (62J05) Estimation in multivariate analysis (62H12) Ridge regression; shrinkage estimators (Lasso) (62J07) Minimax procedures in statistical decision theory (62C20)

Abstract: We study asymptotic minimax problems for estimating a $d$-dimensional regression parameter over spheres of growing dimension ($doinfty$). Assuming that the data follows a linear model with Gaussian predictors and errors, we show that ridge regression is asymptotically minimax and derive new closed form expressions for its asymptotic risk under squared-error loss. The asymptotic risk of ridge regression is closely related to the Stieltjes transform of the Marv{c}enko-Pastur distribution and the spectral distribution of the predictors from the linear model. Adaptive ridge estimators are also proposed (which adapt to the unknown radius of the sphere) and connections with equivariant estimation are highlighted. Our results are mostly relevant for asymptotic settings where the number of observations, $n$, is proportional to the number of predictors, that is, $d/nohoin(0,infty)$.

Recommendations

• Asymptotic Aspects of Ordinary Ridge Regression
• Ridge regression estimators in the linear regression models with non-spherical errors
• Theory of ridge regression estimation with applications
• scientific article; zbMATH DE number 3879946
• Geometry and properties of generalized ridge regression in high dimensions
• Ridge Estimation under the Stochastic Restriction
• High-dimensional asymptotics of prediction: ridge regression and classification

Cites work

• scientific article; zbMATH DE number 3886886 (Why is no real title available?)
• scientific article; zbMATH DE number 3156765 (Why is no real title available?)
• scientific article; zbMATH DE number 1048002 (Why is no real title available?)
• scientific article; zbMATH DE number 3352023 (Why is no real title available?)
• A dozen de Finetti-style results in search of a theory
• A proof of the Fisher information inequality via a data processing argument
• Adaptive prediction and estimation in linear regression with infinitely many parameters.
• Admissible Estimators, Recurrent Diffusions, and Insoluble Boundary Value Problems
• Amenability: A survey for statistical applications of hunt-stein and related conditions on groups
• An ancillarity paradox which appears in multiple linear regression
• Conditional predictive inference post model selection
• Convergence Rates of Spectral Distributions of Large Sample Covariance Matrices
• Convergence rate of expected spectral distributions of large random matrices. II: Sample covariance matrices
• DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES
• Estimation of a multivariate mean with constraints on the norm
• How Many Variables Should be Entered in a Regression Equation?
• Inadmissibility of maximum likelihood estimators in some multiple regression problems with three or more independent variables
• Inequalities for the gamma function
• Information inequalities for the Bayes risk
• Information inequality bounds on the minimax risk (with an application to nonparametric regression)
• Local operator theory, random matrices and Banach spaces.
• Minimax Rates of Estimation for High-Dimensional Linear Regression Over $\ell_q$-Balls
• Minimax estimation of the mean of a normal distribution when the parameter space is restricted
• Minimax risk over \(l_ p\)-balls for \(l_ q\)-error
• Modified Bessel functions and their applications in probability and statistics
• Nearly unbiased variable selection under minimax concave penalty
• On minimax filtering over ellipsoids
• Optimal equivariant prediction for high-dimensional linear models with arbitrary predictor covariance
• Optimal filtering of square-integrable signals in Gaussian noise
• Optimal prediction for linear regression with infinitely many parameters.
• Quasilinear estimates of signals in \(L_ 2\)
• Rate minimaxity of the Lasso and Dantzig selector for the \(l_{q}\) loss in \(l_{r}\) balls
• Ridge Regression and James-Stein Estimation: Review and Comments
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Scaled sparse linear regression
• Simultaneous analysis of Lasso and Dantzig selector
• Some inequalities satisfied by the quantities of information of Fisher and Shannon
• Sparsity oracle inequalities for the Lasso
• Spectrum estimation for large dimensional covariance matrices using random matrix theory
• Square-root lasso: pivotal recovery of sparse signals via conic programming
• Statistical decision theory and Bayesian analysis. 2nd ed
• Strong convergence of the empirical distribution of eigenvalues of large dimensional random matrices
• The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
• Variance estimation in high-dimensional linear models

Cited in

(20)

• Ridge Regression Under Dense Factor Augmented Models
• scientific article; zbMATH DE number 7750670 (Why is no real title available?)
• High-dimensional asymptotics of prediction: ridge regression and classification
• Adaptive estimation in multivariate response regression with hidden variables
• High-Dimensional Factor Regression for Heterogeneous Subpopulations
• Universality of regularized regression estimators in high dimensions
• On the asymptotic risk of ridge regression with many predictors
• A Universal Trade-off Between the Model Size, Test Loss, and Training Loss of Linear Predictors
• In Nonparametric and High-Dimensional Models, Bayesian Ignorability is an Informative Prior
• On the accuracy in high-dimensional linear models and its application to genomic selection
• Asymptotic efficiency of ridge estimator in linear and semiparametric linear models
• Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime*
• Asymptotic properties on high-dimensional multivariate regression M-estimation
• On the behavior of the risk of a Lasso-type estimator
• Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
• Significance testing in non-sparse high-dimensional linear models
• Cleaning large correlation matrices: tools from random matrix theory
• Smoothly varying regularization
• Surprises in high-dimensional ridgeless least squares interpolation
• WONDER: weighted one-shot distributed ridge regression in high dimensions

This page was built for publication: Ridge regression and asymptotic minimax estimation over spheres of growing dimension

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5963493)