Mixtures, envelopes and hierarchical duality

DOI10.1111/RSSB.12130zbMATH Open1414.62056arXiv1406.0177OpenAlexW1827621890MaRDI QIDQ5378366FDOQ5378366

Nicholas G. Polson, James G. Scott

Publication date: 12 June 2019

Published in: Journal of the Royal Statistical Society Series B: Statistical Methodology (Search for Journal in Brave)

Abstract: We develop a connection between mixture and envelope representations of objective functions that arise frequently in statistics. We refer to this connection using the term "hierarchical duality." Our results suggest an interesting and previously under-exploited relationship between marginalization and profiling, or equivalently between the Fenchel--Moreau theorem for convex functions and the Bernstein--Widder theorem for Laplace transforms. We give several different sets of conditions under which such a duality result obtains. We then extend existing work on envelope representations in several ways, including novel generalizations to variance-mean models and to multivariate Gaussian location models. This turns out to provide an elegant missing-data interpretation of the proximal gradient method, a widely used algorithm in machine learning. We show several statistical applications in which the proposed framework leads to easily implemented algorithms, including a robust version of the fused lasso, nonlinear quantile regression via trend filtering, and the binomial fused double Pareto model. Code for the examples is available on GitHub at https://github.com/jgscott/hierduals.

Full work available at URL: https://arxiv.org/abs/1406.0177

Recommendations

• Proximal algorithms in statistics and machine learning
• Foundations for envelope models and methods
• Duality results and proximal solutions of the Huber \(M\)-estimator problem
• \(\ell_{1}\)-penalization for mixture regression models
• Unifying Divergence Minimization and Statistical Inference Via Convex Duality

zbMATH Keywords

Bayesian inferencepenalized likelihoodenvelopesGaussian mixturesconvex dualityvariational methodsmaximum \textit{a posteriori} estimation

Mathematics Subject Classification ID

Point estimation (62F10) Bayesian inference (62F15) Nonparametric estimation (62G05) Software, source code, etc. for problems pertaining to statistics (62-04) Ridge regression; shrinkage estimators (Lasso) (62J07) Minimax procedures in statistical decision theory (62C20) Statistical distribution theory (62Exx)

Cited In (5)

• Horseshoe Regularisation for Machine Learning in Complex and Deep Models¹
• Proximal algorithms in statistics and machine learning
• Bayesian \(l_0\)-regularized least squares
• The horseshoe-like regularization for feature subset selection
• Weighted Bayesian bootstrap for scalable posterior distributions

Uses Software

• GitHub

This page was built for publication: Mixtures, envelopes and hierarchical duality

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5378366)