Gaussian process regression in the flat limit

DOI10.1214/23-AOS2336arXiv2201.01074MaRDI QIDQ6183872FDOQ6183872

Authors: Simon Barthelmé, Pierre-Olivier Amblard, N. Tremblay, Konstantin Usevich 

Publication date: 4 January 2024

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: Gaussian process (GP) regression is a fundamental tool in Bayesian statistics. It is also known as kriging and is the Bayesian counterpart to the frequentist kernel ridge regression. Most of the theoretical work on GP regression has focused on a large-$n$ asymptotics, characterising the behaviour of GP regression as the amount of data increases. Fixed-sample analysis is much more difficult outside of simple cases, such as locations on a regular grid. In this work we perform a fixed-sample analysis that was first studied in the context of approximation theory by Driscoll & Fornberg (2002), called the "flat limit". In flat-limit asymptotics, the goal is to characterise kernel methods as the length-scale of the kernel function tends to infinity, so that kernels appear flat over the range of the data. Surprisingly, this limit is well-defined, and displays interesting behaviour: Driscoll & Fornberg showed that radial basis interpolation converges in the flat limit to polynomial interpolation, if the kernel is Gaussian. Leveraging recent results on the spectral behaviour of kernel matrices in the flat limit, we study the flat limit of Gaussian process regression. Results show that Gaussian process regression tends in the flat limit to (multivariate) polynomial regression, or (polyharmonic) spline regression, depending on the kernel. Importantly, this holds for both the predictive mean and the predictive variance, so that the posterior predictive distributions become equivalent. Our results have practical consequences: for instance, they show that optimal GP predictions in the sense of leave-one-out loss may occur at very large length-scales, which would be invisible to current implementations because of numerical difficulties.

Full work available at URL: https://arxiv.org/abs/2201.01074

Recommendations

• Understanding Gaussian Process Regression Using the Equivalent Kernel
• Gaussian processes and limiting linear models
• Asymptotic analysis of the learning curve for Gaussian process regression

zbMATH Keywords

splinesGaussian processesmultivariate polynomialsflat limit

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Numerical smoothing, curve fitting (65D10)

Cites Work

• Semiparametric Regression
• Gaussian Markov Random Fields
• Generalized additive models. An introduction with R.
• The elements of statistical learning. Data mining, inference, and prediction
• Stable Computations with Gaussian Radial Basis Functions
• A new look at the statistical model identification
• Interpolation of spatial data. Some theory for kriging
• Gaussian processes for machine learning.
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach
• A taxonomy of global optimization methods based on response surfaces
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Scattered Data Approximation
• Smoothing by spline functions.
• Interpolation of scattered data: distance matrices and conditionally positive definite functions
• Linear smoothers and additive models
• Rates of contraction of posterior distributions based on Gaussian process priors
• Title not available (Why is that?)
• Interpolation in the limit of increasingly flat radial basis functions
• Interpolation of spatial data -- a stochastic or a deterministic problem?
• A Stable Algorithm for Flat Radial Basis Functions on a Sphere
• Theoretical and computational aspects of multivariate interpolation with increasingly flat radial basis functions
• Multivariate interpolation with increasingly flat radial basis functions of finite smoothness
• Multivariate interpolation by polynomials and radial basis functions
• Multivariate interpolation at arbitrary points made simple
• Polynomial interpolation in several variables: lattices, differences, and ideals
• From Stein's unbiased risk estimates to the method of generalized cross- validation
• Efficient nonparametric Bayesian inference for \(X\)-ray transforms
• Bayesian Probabilistic Numerical Methods
• Spectral properties of kernel matrices in the flat limit
• Title not available (Why is that?)
• Expectation Propagation in the Large Data Limit
• Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
• Statistical guarantees for Bayesian uncertainty quantification in nonlinear inverse problems with Gaussian process priors
• Determinantal point processes in the flat limit
• Determinantal Point Processes Implicitly Regularize Semiparametric Regression Problems
• Extended L-ensembles: a new representation for determinantal point processes

Cited In (4)

• Gaussian processes and limiting linear models
• Title not available (Why is that?)
• Understanding Gaussian Process Regression Using the Equivalent Kernel
• Weighted leave-one-out cross validation

This page was built for publication: Gaussian process regression in the flat limit

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6183872)