ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

DOI10.1007/978-3-319-41589-5_13MaRDI QIDQ5350485zbMATH OpenOpenAlexFDO

Authors Yue Hu, Eric C. Chi, Genevera I. Allen

Publication date 1 September 2017

Published in Splitting Methods in Communication, Imaging, Science, and Engineering (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1504.06637

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Linear regression; mixed models (62J05) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: Optimization approaches based on operator splitting are becoming popular for solving sparsity regularized statistical machine learning models. While many have proposed fast algorithms to solve these problems for a single regularization parameter, conspicuously less attention has been given to computing regularization paths, or solving the optimization problems over the full range of regularization parameters to obtain a sequence of sparse models. In this chapter, we aim to quickly approximate the sequence of sparse models associated with regularization paths for the purposes of statistical model selection by using the building blocks from a classical operator splitting method, the Alternating Direction Method of Multipliers (ADMM). We begin by proposing an ADMM algorithm that uses warm-starts to quickly compute the regularization path. Then, by employing approximations along this warm-starting ADMM algorithm, we propose a novel concept that we term the ADMM Algorithmic Regularization Path. Our method can quickly outline the sequence of sparse models associated with the regularization path in computational time that is often less than that of using the ADMM algorithm to solve the problem at a single regularization parameter. We demonstrate the applicability and substantial computational savings of our approach through three popular examples, sparse linear regression, reduced-rank multi-task learning, and convex clustering.

Recommendations

• ADMM algorithmic regularization paths for high-dimensional sparse precision matrix estimation
• A sparsity preserving stochastic gradient methods for sparse regression
• On sparsity inducing regularization methods for machine learning
• Pathwise coordinate optimization for sparse learning: algorithm and theory
• Stochastic alternating direction method of multipliers for structured regularization
• Adaptive Randomized Coordinate Descent for Sparse Systems: Lasso and Greedy Algorithms
• Nonparametric sparsity and regularization
• Converting ADMM to a proximal gradient for efficient sparse estimation
• Alternating direction method of multipliers for machine learning
• Alternating direction method of multipliers for sparse principal component analysis

Cited in

(9)

• Adaptive Minimax Estimation over Sparse \ell_q-Hulls
• Pathwise coordinate optimization for sparse learning: algorithm and theory
• Global convergence of ADMM in nonconvex nonsmooth optimization
• Multivariate response regression with low-rank and generalized sparsity
• Adaptive Randomized Coordinate Descent for Sparse Systems: Lasso and Greedy Algorithms
• Poisson noise removal based on non-convex hybrid regularizers
• The sparse dynamic factor model: a regularised quasi-maximum likelihood approach
• ADMM algorithmic regularization paths for high-dimensional sparse precision matrix estimation
• Fourier transform sparse inverse regression estimators for sufficient variable selection

This page was built for publication: ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5350485)