Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions

DOI10.1137/20M1315968MaRDI QIDQ5119635zbMATH OpenOpenAlexFDO

Authors Toni Karvonen, George Wynne, Filip Tronarp, Chris J. Oates, Simo Särkkä

Publication date 31 August 2020

Published in SIAM/ASA Journal on Uncertainty Quantification (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/2001.10965

zbMATH Keywords

Bayesian cubaturecredible setsmodel misspecificationnonparametric regressionscattered data approximation

Mathematics Subject Classification ID

Hilbert spaces with reproducing kernels (= (proper) functional Hilbert spaces, including de Branges-Rovnyak and other structured spaces) (46E22) Gaussian processes (60G15) Asymptotic properties of nonparametric inference (62G20) Numerical interpolation (65D05) Reasoning under uncertainty in the context of artificial intelligence (68T37)

Abstract: Despite the ubiquity of the Gaussian process regression model, few theoretical results are available that account for the fact that parameters of the covariance kernel typically need to be estimated from the dataset. This article provides one of the first theoretical analyses in the context of Gaussian process regression with a noiseless dataset. Specifically, we consider the scenario where the scale parameter of a Sobolev kernel (such as a Mat'{e}rn kernel) is estimated by maximum likelihood. We show that the maximum likelihood estimation of the scale parameter alone provides significant adaptation against misspecification of the Gaussian process model in the sense that the model can become "slowly" overconfident at worst, regardless of the difference between the smoothness of the data-generating function and that expected by the model. The analysis is based on a combination of techniques from nonparametric regression and scattered data interpolation. Empirical results are provided in support of the theoretical findings.

Recommendations

• Properties of the Bayesian parameter estimation of a regression based on Gaussian processes
• Can we trust Bayesian uncertainty quantification from Gaussian process priors with squared exponential covariance kernel?
• On the inference of applying Gaussian process modeling to a deterministic function
• Asymptotic properties of a maximum likelihood estimator with data from a Gaussian process
• Advanced Lectures on Machine Learning

Cites work

• A unified theory of radial basis functions. Native Hilbert spaces for radial basis functions. II
• An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach
• An extension of a bound for functions in Sobolev spaces, with applications to \((m, s)\)-spline interpolation and smoothing
• Approximate interpolation with applications to selecting smoothing parameters
• Approximation theory and algorithms for data analysis
• Asymptotic analysis of covariance parameter estimation for Gaussian processes in the misspecified case
• Average-case analysis of numerical problems
• Bayes-Hermite quadrature
• Bayesian calibration of computer models. (With discussion)
• Bayesian Probabilistic Numerical Methods
• Convergence analysis of deterministic kernel-based quadrature rules in misspecified settings
• Convergence rates of efficient global optimization algorithms
• Convergence types and rates in generic Karhunen-Loève expansions with applications to sample path properties
• Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification
• Cross-validation estimation of covariance parameters under fixed-domain asymptotics
• Design and analysis of computer experiments. With comments and a rejoinder by the authors
• Deterministic and stochastic error bounds in numerical analysis
• Empirical Bayes scaling of Gaussian priors in the white noise model
• Error estimates and condition numbers for radial basis function interpolation
• Extension of sampling inequalities to Sobolev semi-norms of fractional order and derivative data
• Fast automatic Bayesian cubature using lattice sampling
• Frequentist coverage of adaptive nonparametric Bayesian credible sets
• Gaussian measure in Hilbert space and applications in numerical analysis
• Gaussian processes for machine learning.
• scientific article; zbMATH DE number 4138299 (Why is no real title available?)
• scientific article; zbMATH DE number 45971 (Why is no real title available?)
• scientific article; zbMATH DE number 1223843 (Why is no real title available?)
• scientific article; zbMATH DE number 1098876 (Why is no real title available?)
• scientific article; zbMATH DE number 6827443 (Why is no real title available?)
• scientific article; zbMATH DE number 5055767 (Why is no real title available?)
• Improved error bounds for scattered data interpolation by radial basis functions
• Information rates of nonparametric Gaussian process methods
• Interpolation of spatial data -- a stochastic or a deterministic problem?
• Interpolation of spatial data. Some theory for kriging
• Kernel-based approximation methods using MATLAB
• Maximum Likelihood Estimation for a Smooth Gaussian Random Field Model
• Maximum likelihood estimation for Gaussian processes under inequality constraints
• Mercer's theorem on general domains: on the interaction between measures, kernels, and RKHSs
• On the equivalence between kernel quadrature rules and random feature expansions
• On the marginal likelihood and cross-validation
• Probabilistic integration: a role in statistical computation?
• Probabilistic numerics and uncertainty in computations
• Regularity of the sample paths of a general second order random field
• Sampling inequalities for infinitely smooth functions, with applications to interpolation and machine learning
• Scattered Data Approximation
• Simultaneous inference for function-valued parameters: a fast and fair approach
• Sobolev error estimates and a Bernstein inequality for scattered data interpolation via radial basis functions
• Spline smoothing with an estimated order parameter
• Stability of kernel-based interpolation
• Stochastic processes with sample paths in reproducing kernel Hilbert spaces
• Superconvergence of kernel-based interpolation
• The reproducing kernel Hilbert space structure of the sample paths of a Gaussian process
• Theory of function spaces. III
• Tractability of multivariate problems. Volume I: Linear information

Cited in

(29)

• Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification
• Bayesian ODE solvers: the maximum a posteriori estimate
• Bayesian numerical methods for nonlinear partial differential equations
• On the inference of applying Gaussian process modeling to a deterministic function
• Sampling based approximation of linear functionals in reproducing kernel Hilbert spaces
• Distribution-free uncertainty quantification for kernel methods by gradient perturbations
• Exploiting Hessian matrix and trust-region algorithm in hyperparameters estimation of Gaussian process
• Small sample spaces for Gaussian processes
• A matrix-free approach for solving the parametric Gaussian process maximum likelihood problem
• Convergence guarantees for Gaussian process means with misspecified likelihoods and smoothness
• Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
• Maximum Likelihood Estimation for a Smooth Gaussian Random Field Model
• Can Gaussian Process Regression Be Made Robust Against Model Mismatch?
• Fixed-domain asymptotics of the maximum likelihood estimator and the Gaussian process approach for deterministic models
• Can we trust Bayesian uncertainty quantification from Gaussian process priors with squared exponential covariance kernel?
• Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation
• Discrimination between Gaussian process models: active learning and static constructions
• Parameter Selection in Gaussian Process Interpolation: An Empirical Study of Selection Criteria
• Error bounds and the asymptotic setting in kernel-based approximation
• Weighted leave-one-out cross validation
• Convergence rates of non-stationary and deep Gaussian process regression
• Two-stage estimators for spatial confounding with point-referenced data
• Probabilistic Richardson extrapolation
• Smoothness estimation for Whittle-Matérn processes on closed Riemannian manifolds
• Error analysis for a statistical finite element method
• Comparing scale parameter estimators for Gaussian process interpolation with the Brownian motion prior: leave-one-out cross validation and maximum likelihood
• An asymptotic study of the joint maximum likelihood estimation of the regularity and the amplitude parameters of a periodized Matérn model
• Relaxed Gaussian process interpolation: a goal-oriented approach to Bayesian optimization
• A theoretical framework of the scaled Gaussian stochastic process in prediction and calibration

Describes a project that uses

Uses Software

• GaussQR

This page was built for publication: Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5119635)