Anomalies in the Foundations of Ridge Regression

DOI10.1111/J.1751-5823.2007.00041.XzbMATH Open1206.62129arXivmath/0703551OpenAlexW2087871315WikidataQ57447150 ScholiaQ57447150MaRDI QIDQ3182012FDOQ3182012

Donald E. Ramirez, Donald R. Jensen

Publication date: 1 October 2009

Published in: International Statistical Review (Search for Journal in Brave)

Abstract: Anomalies persist in the foundations of ridge regression as set forth in Hoerl and Kennard (1970) and subsequently. Conventional ridge estimators and their properties do not follow on constraining lengths of solution vectors using LaGrange's method, as claimed. Estimators so constrained have singular distributions; the proposed solutions are not necessarily minimizing; and heretofore undiscovered bounds are exhibited for the ridge parameter. None of the considerable literature on estimation, prediction, cross--validation, choice of ridge parameter, and related issues, collectively known as ridge regression, is consistent with constrained optimization, nor with corresponding inequality constraints. The problem is traced to a misapplication of LaGrange's principle, failure to recognize the singularity of distributions, and misplaced links between constraints and the ridge parameter. Other principles, based on condition numbers, are seen to validate both conventional ridge and surrogate ridge regression to be defined. Numerical studies illustrate that ridge analysis often exhibits some of the same pathologies it is intended to redress.

Full work available at URL: https://arxiv.org/abs/math/0703551

zbMATH Keywords

constrained optimizationalternative foundationsincomplete use of LaGrange's methodnon-singular distributions

Mathematics Subject Classification ID

Estimation in multivariate analysis (62H12) Ridge regression; shrinkage estimators (Lasso) (62J07)

Cites Work

• Matrix Analysis
• Some Comments on C P
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• Ridge Regression: Biased Estimation for Nonorthogonal Problems
• Ridge Regression in Practice
• Tolerance and Condition in Regression Computations
• A Biometrics Invited Paper. The Analysis and Selection of Variables in Linear Regression
• Title not available (Why is that?)
• The Relationship between Variable Selection and Data Agumentation and a Method for Prediction
• Title not available (Why is that?)
• Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation
• Inequalities: theory of majorization and its applications
• Ridge regression:some simulations
• Solving Systems of Linear Equations With a Positive Definite, Symmetric, but Possibly Ill-Conditioned Matrix
• The Acceptability of Regression Solutions: Another Look at Computational Accuracy
• The generalized ridge estimator and improved adjustments for regression parameters
• Minimax linear, ridge and shrunken estimators for linear parameters
• The Effect of Errors in the Independent Variables in Linear Regression
• "Ridge Analysis" of Response Surfaces
• Characterization of Ridge Trace Behavior
• A Simulation Study of Alternatives to Ordinary Least Squares

Cited In (16)

• A note about the corrected VIF
• On mitigating collinearity through mixtures
• Revision: variance inflation in regression
• Mitigating collinearity in linear regression models using ridge, surrogate and raised estimators
• On the practice of rescaling covariates
• Anomalies in the foundations of ridge regression: some clarifications
• Enhanced ridge regressions
• Nonparametric Principal Components Regression
• Collinearity: revisiting the variance inflation factor in ridge regression
• Designs enhancing Fisher information
• The red indicator and corrected VIFs in generalized linear models
• Confronting collinearity in environmental regression models: evidence from world data
• A Note on Collinearity Diagnostics and Centering
• Standardization of Variables and Collinearity Diagnostic in Ridge Regression
• Penalized regression with ordinal predictors
• The coefficient of determination in the ridge regression

   Recommendations

• Title not available (Why is that?) 👍 👎
• Concentration reversals in ridge regression 👍 👎
• Ridge Regression — 1980: Advances, Algorithms, and Applications 👍 👎
• Condition Numbers and Minimax Ridge Regression Estimators 👍 👎
• Tracking MSE efficiencies in ridge regression 👍 👎

This page was built for publication: Anomalies in the Foundations of Ridge Regression

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q3182012)