A generic path algorithm for regularized statistical estimation

DOI10.1080/01621459.2013.864166zbMATH Open1367.62223arXiv1201.3571OpenAlexW2146031909WikidataQ42905187 ScholiaQ42905187MaRDI QIDQ4975409FDOQ4975409

Authors: Hua Zhou, Yichao Wu 

Publication date: 4 August 2017

Published in: Journal of the American Statistical Association (Search for Journal in Brave)

Abstract: Regularization is widely used in statistics and machine learning to prevent overfitting and gear solution towards prior information. In general, a regularized estimation problem minimizes the sum of a loss function and a penalty term. The penalty term is usually weighted by a tuning parameter and encourages certain constraints on the parameters to be estimated. Particular choices of constraints lead to the popular lasso, fused-lasso, and other generalized $l_1$ penalized regression methods. Although there has been a lot of research in this area, developing efficient optimization methods for many nonseparable penalties remains a challenge. In this article we propose an exact path solver based on ordinary differential equations (EPSODE) that works for any convex loss function and can deal with generalized $l_1$ penalties as well as more complicated regularization such as inequality constraints encountered in shape-restricted regressions and nonparametric density estimation. In the path following process, the solution path hits, exits, and slides along the various constraints and vividly illustrates the tradeoffs between goodness of fit and model parsimony. In practice, the EPSODE can be coupled with AIC, BIC, $C_p$ or cross-validation to select an optimal tuning parameter. Our applications to generalized $l_1$ regularized generalized linear models, shape-restricted regressions, Gaussian graphical models, and nonparametric density estimation showcase the potential of the EPSODE algorithm.

Full work available at URL: https://arxiv.org/abs/1201.3571

Recommendations

• The solution path of the generalized lasso
• Piecewise linear regularized solution paths
• An ordinary differential equation-based solution path algorithm
• A New Algorithm in Maximum Likelihood Estimation for Generalized Linear Models
• Efficient global approximation of generalized nonlinear \(\ell _{1}\)-regularized solution paths and its applications

zbMATH Keywords

LassoGaussian graphical modelgeneralized linear modelregularizationquasi-likelihoodordinary differential equationssolution pathlog-concave density estimationshape restricted regression

Mathematics Subject Classification ID

Density estimation (62G07) Ridge regression; shrinkage estimators (Lasso) (62J07) Generalized linear models (logistic models) (62J12)

Cites Work

• Least angle regression. (With discussion)
• Ideal spatial adaptation by wavelet shrinkage
• Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties
• Limit distribution theory for maximum likelihood estimation of a log-concave density
• Title not available (Why is that?)
• Sparsity and Smoothness Via the Fused Lasso
• Title not available (Why is that?)
• A unified framework for high-dimensional analysis of \(M\)-estimators with decomposable regularizers
• The solution path of the generalized lasso
• Inference and modeling with log-concave distributions
• Sparse inverse covariance estimation with the graphical lasso
• Additive logistic regression: a statistical view of boosting. (With discussion and a rejoinder by the authors)
• Piecewise linear regularized solution paths
• Title not available (Why is that?)
• Title not available (Why is that?)
• A new approach to variable selection in least squares problems
• An active set algorithm to estimate parameters in generalized linear models with ordered predictors
• On the ``degrees of freedom of the lasso
• Numerical analysis for statisticians
• $\ell_1$ Trend Filtering
• Detecting the Presence of Mixing with Multiscale Maximum Likelihood
• Quantile Regression in Reproducing Kernel Hilbert Spaces
• An ordinary differential equation-based solution path algorithm
• An improved model averaging scheme for logistic regression
• Parametrically guided generalised additive models with application to mergers and acquisitions data

Cited In (18)

• An ordinary differential equation-based solution path algorithm
• Primal path algorithm for compositional data analysis
• Path-following methods for maximum a posteriori estimators in Bayesian hierarchical models: how estimates depend on hyperparameters
• Dynamic Visualization and Fast Computation for Convex Clustering via Algorithmic Regularization
• Efficient regularized regression with \(L_0\) penalty for variable selection and network construction
• On the structure of regularization paths for piecewise differentiable regularization terms
• Algorithms for Fitting the Constrained Lasso
• Efficient computation for differential network analysis with applications to quadratic discriminant analysis
• Title not available (Why is that?)
• Model Selection for High-Dimensional Quadratic Regression via Regularization
• L 1-Regularization Path Algorithm for Generalized Linear Models
• Efficient global approximation of generalized nonlinear \(\ell _{1}\)-regularized solution paths and its applications
• Global least squares path modeling: a full-information alternative to partial least squares path modeling
• Weak convergence of the regularization path in penalized M-estimation
• Path following in the exact penalty method of convex programming
• Title not available (Why is that?)
• LARS-type algorithm for group Lasso
• Solution path for quantile regression with epsilon-insensitive loss in a reproducing kernel Hilbert space

This page was built for publication: A generic path algorithm for regularized statistical estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4975409)