Laplace approximation and natural gradient for Gaussian process regression with heteroscedastic Student-t model

DOI10.1007/S11222-018-9836-0MaRDI QIDQ2329797zbMATH OpenOpenAlexFDO

Authors Marcelo Hartmann, Jarno Vanhatalo

Publication date 18 October 2019

Published in Statistics and Computing (Search for Journal in Brave)

Copyright license Creative Commons Attribution 4.0 International

Full work available at URL https://arxiv.org/abs/1712.07437

zbMATH Keywords

approximate inferenceFisher information matrixGaussian processesheteroscedastic noiseLaplace approximationlocation-scale regressionnatural gradientRiemannian metricStudent-\(t\) model

Mathematics Subject Classification ID

Gaussian processes (60G15) Statistical aspects of information-theoretic topics (62B10) Parametric hypothesis testing (62F03) General nonlinear regression (62J02) Statistics on manifolds (62R30) Monte Carlo methods (65C05)

Abstract: This paper considers the Laplace method to derive approximate inference for the Gaussian process (GP) regression in the location and scale parameters of the Student-t probabilistic model. This allows both mean and variance of the data to vary as a function of covariates with the attractive feature that the Student-t model has been widely used as a useful tool for robustifying data analysis. The challenge in the approximate inference for the GP regression with the Student-t probabilistic model, lies in the analytical intractability of the posterior distribution and the lack of concavity of the log-likelihood function. We present the natural gradient adaptation for the estimation process which primarily relies on the property that the Student-t model naturally has orthogonal parametrization with respect to the location and scale paramaters. Due to this particular property of the model, we also introduce an alternative Laplace approximation by using the Fisher information matrix in place of the Hessian matrix of the negative log-likelihood function. According to experiments this alternative approximation provides very similar posterior approximations and predictive performance when compared to the traditional Laplace approximation. We also compare both of these Laplace approximations with the Monte Carlo Markov Chain (MCMC) method. Moreover, we compare our heteroscedastic Student-t model and the GP regression with the heteroscedastic Gaussian model. We also discuss how our approach can improve the inference algorithm in cases where the probabilistic model assumed for the data is not log-concave.

Recommendations

• Robust Gaussian process regression with a Student-t likelihood
• The Variational Gaussian Approximation Revisited
• Laplace approximation for logistic Gaussian process density estimation and regression
• Student-t Process Regression with Dependent Student-t Noise
• scientific article; zbMATH DE number 774847

Cites work

• A survey of Bayesian predictive methods for model assessment, selection and comparison
• Accurate Approximations for Posterior Moments and Marginal Densities
• Approximate Bayes factors and accounting for model uncertainty in generalised linear models
• Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion)
• Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes
• Approximations for binary Gaussian process classification
• Assessing approximate inference for binary Gaussian process classification.
• Bayes Factors
• Choice of basis for Laplace approximation
• Distributed Bayesian learning with stochastic natural gradient expectation propagation and the posterior server
• Fully Exponential Laplace Approximations to Expectations and Variances of Nonpositive Functions
• Gaussian processes for machine learning (GPML) toolbox
• Gaussian processes for machine learning.
• GPstuff: Bayesian modeling with Gaussian processes
• Hedonic housing prices and the demand for clean air
• scientific article; zbMATH DE number 3426675 (Why is no real title available?)
• scientific article; zbMATH DE number 3121201 (Why is no real title available?)
• scientific article; zbMATH DE number 3848401 (Why is no real title available?)
• scientific article; zbMATH DE number 3905641 (Why is no real title available?)
• scientific article; zbMATH DE number 3980241 (Why is no real title available?)
• scientific article; zbMATH DE number 3656961 (Why is no real title available?)
• scientific article; zbMATH DE number 3667770 (Why is no real title available?)
• scientific article; zbMATH DE number 46726 (Why is no real title available?)
• scientific article; zbMATH DE number 3583145 (Why is no real title available?)
• scientific article; zbMATH DE number 409719 (Why is no real title available?)
• scientific article; zbMATH DE number 578421 (Why is no real title available?)
• scientific article; zbMATH DE number 1941665 (Why is no real title available?)
• scientific article; zbMATH DE number 3998953 (Why is no real title available?)
• scientific article; zbMATH DE number 1865509 (Why is no real title available?)
• scientific article; zbMATH DE number 775283 (Why is no real title available?)
• scientific article; zbMATH DE number 816038 (Why is no real title available?)
• scientific article; zbMATH DE number 7306852 (Why is no real title available?)
• scientific article; zbMATH DE number 3211315 (Why is no real title available?)
• scientific article; zbMATH DE number 3054872 (Why is no real title available?)
• scientific article; zbMATH DE number 3106666 (Why is no real title available?)
• Information-geometric optimization algorithms: a unifying picture via invariance principles
• Multivariate adaptive regression splines
• Multivariate Student-t regression models: Pitfalls and inference
• Newton's method and its use in optimization
• Newton-Raphson and Related Algorithms for Maximum Likelihood Variance Component Estimation
• Objective Bayesian analysis for the Student-t regression model
• On the comparison of Fisher information of the Weibull and GE distributions
• Pattern recognition and machine learning.
• Penalising model component complexity: a principled, practical approach to constructing priors
• Posterior expectations for large observations
• Posterior property of Student-t linear regression model using objective priors
• Prior distributions for variance parameters in hierarchical models (Comment on article by Browne and Draper)
• Riemann manifold Langevin and Hamiltonian Monte Carlo methods. With discussion and authors' reply
• Robust diagnostic regression analysis
• Robust Gaussian process regression with a Student-t likelihood
• Some diagnostics of maximum likelihood and posterior nonnormality
• THE PROBABLE ERROR OF A MEAN
• Understanding predictive information criteria for Bayesian models

Cited in

(7)

• Identification of Gaussian process with switching noise mode and missing data
• Estimation and prediction of a generalized mixed-effects model with \(t\)-process for longitudinal correlated binary data
• Mixture of multivariate Gaussian processes for classification of irregularly sampled satellite image time-series
• Robust Gaussian process regression with a Student-t likelihood
• Robust variational Gaussian process regression for count data with the trimmed marginal likelihood
• Orthogonal parametrisations of extreme-value distributions
• Forecasts of composite real estate price indices through Gaussian process regressions

Describes a project that uses

Uses Software

• GPstuff
• PRMLT
• GPML

This page was built for publication: Laplace approximation and natural gradient for Gaussian process regression with heteroscedastic Student-\(t\) model

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2329797)