Sparse regression learning by aggregation and Langevin Monte-Carlo
From MaRDI portal
(Redirected from Publication:439987)
Abstract: We consider the problem of regression learning for deterministic design and independent random errors. We start by proving a sharp PAC-Bayesian type bound for the exponentially weighted aggregate (EWA) under the expected squared empirical loss. For a broad class of noise distributions the presented bound is valid whenever the temperature parameter of the EWA is larger than or equal to , where is the noise variance. A remarkable feature of this result is that it is valid even for unbounded regression functions and the choice of the temperature parameter depends exclusively on the noise level. Next, we apply this general bound to the problem of aggregating the elements of a finite-dimensional linear space spanned by a dictionary of functions . We allow to be much larger than the sample size but we assume that the true regression function can be well approximated by a sparse linear combination of functions . Under this sparsity scenario, we propose an EWA with a heavy tailed prior and we show that it satisfies a sparsity oracle inequality with leading constant one. Finally, we propose several Langevin Monte-Carlo algorithms to approximately compute such an EWA when the number of aggregated functions can be large. We discuss in some detail the convergence of these algorithms and present numerical experiments that confirm our theoretical findings.
Recommendations
- Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
- On the exponentially weighted aggregate with the Laplace prior
- Aggregated estimators and empirical complexity for least square regression
- PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting
- Aggregation by Exponential Weighting and Sharp Oracle Inequalities
Cites work
- scientific article; zbMATH DE number 5957408 (Why is no real title available?)
- scientific article; zbMATH DE number 5544465 (Why is no real title available?)
- scientific article; zbMATH DE number 4020069 (Why is no real title available?)
- Aggregating regression procedures to improve performance
- Aggregation and Sparsity Via ℓ1 Penalized Least Squares
- Aggregation by Exponential Weighting and Sharp Oracle Inequalities
- Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
- Aggregation for Gaussian regression
- An Empirical Bayesian Strategy for Solving the Simultaneous Sparse Approximation Problem
- Bayesian inference and optimal design for the sparse linear model
- Empirical Bayes selection of wavelet thresholds
- Fast learning rates in statistical inference through aggregation
- Graph selection with GGMselect
- High-dimensional generalized linear models and the lasso
- High-dimensional graphs and variable selection with the Lasso
- How to use expert advice
- Information Theory and Mixing Least-Squares Regressions
- Langevin diffusions and Metropolis-Hastings algorithms
- Learning by mirror averaging
- Least angle regression. (With discussion)
- Markov chains and stochastic stability
- Mirror averaging with sparsity priors
- Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies?
- Nearly unbiased variable selection under minimax concave penalty
- Nonlinear estimation over weak Besov spaces and minimax Bayes
- On optimality of Bayesian testimation in the normal means problem
- On the Generalization Ability of On-Line Learning Algorithms
- On the optimality of the aggregate with exponential weights for low temperatures
- Optimal rates and adaptation in the single-index model using aggregation
- PAC-Bayesian bounds for randomized empirical risk minimizers
- PAC-Bayesian stochastic model selection
- Prediction of stochastic sequences
- Prediction, Learning, and Games
- Regularization and Variable Selection Via the Elastic Net
- Sequential prediction of individual sequences under general loss functions
- Simultaneous analysis of Lasso and Dantzig selector
- Sparse recovery in convex hulls via entropy penalization
- Sparse regression learning by aggregation and Langevin Monte-Carlo
- Sparsity oracle inequalities for the Lasso
- Stable recovery of sparse overcomplete representations in the presence of noise
- Statistical learning theory and stochastic optimization. Ecole d'Eté de Probabilitiés de Saint-Flour XXXI -- 2001.
- The Dantzig selector: statistical estimation when \(p\) is much larger than \(n\). (With discussions and rejoinder).
- The sparsity and bias of the LASSO selection in high-dimensional linear regression
- The weighted majority algorithm
- Time-reversible diffusions
Cited in
(38)- Weighted multilevel Langevin simulation of invariant measures
- On the exponentially weighted aggregate with the Laplace prior
- Exponential screening and optimal rates of sparse estimation
- Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
- Probabilistic learning inference of boundary value problem with uncertainties based on Kullback-Leibler divergence under implicit constraints
- PAC-Bayesian bounds for sparse regression estimation with exponential weights
- Mirror averaging with sparsity priors
- Non-convex penalized estimation in high-dimensional models with single-index structure
- Sparse regression learning by aggregation and Langevin Monte-Carlo
- PAC-Bayesian estimation and prediction in sparse additive models
- Prediction error bounds for linear regression with the TREX
- Noisy Monte Carlo: convergence of Markov chains with approximate transition kernels
- Exponential weights in multivariate regression and a low-rankness favoring prior
- Sampling from non-smooth distributions through Langevin diffusion
- Sharp oracle inequalities for low-complexity priors
- The tamed unadjusted Langevin algorithm
- Approximation for the invariant measure with applications for jump processes (convergence in total variation distance)
- Statistical inference in compound functional models
- A reduced-rank approach to predicting multiple binary responses through machine learning
- Approximate models and robust decisions
- Ergodicity of supercritical SDEs driven by \(\alpha \)-stable processes and heavy-tailed sampling
- Sharp oracle inequalities for aggregation of affine estimators
- scientific article; zbMATH DE number 7307484 (Why is no real title available?)
- Sparse recovery under matrix uncertainty
- Simple proof of the risk bound for denoising by exponential weights for asymmetric noise distributions
- Sparse estimation by exponential weighting
- User-friendly Introduction to PAC-Bayes Bounds
- Entropy-based closure for probabilistic learning on manifolds
- Selection of KL neighbourhood in robust Bayesian inference
- Functional inequalities for perturbed measures with applications to log-concave measures and to some Bayesian problems
- Optimal learning with \textit{Q}-aggregation
- PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting
- PAC-Bayesian high dimensional bipartite ranking
- Probabilistic learning on manifolds constrained by nonlinear partial differential equations for small datasets
- A quasi-Bayesian perspective to online clustering
- On Stochastic Gradient Langevin Dynamics with Dependent Data Streams: The Fully Nonconvex Case
- High-dimensional sparse classification using exponential weighting with empirical hinge loss
- Entropic optimal transport is maximum-likelihood deconvolution
This page was built for publication: Sparse regression learning by aggregation and Langevin Monte-Carlo
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q439987)
