Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation

DOI10.1090/MCOM/3649zbMATH Open1472.62012arXiv2005.11375OpenAlexW3155546189MaRDI QIDQ4956916FDOQ4956916

Authors: Yifan Chen, Houman Owhadi, A. M. Stuart 

Publication date: 2 September 2021

Published in: Mathematics of Computation (Search for Journal in Brave)

Abstract: Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Mat'ern-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Mat'ern-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Mat'ern-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.

Full work available at URL: https://arxiv.org/abs/2005.11375

Recommendations

• Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
• Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency
• Hierarchical modeling with gaussian processes
• Properties of the Bayesian parameter estimation of a regression based on Gaussian processes
• scientific article; zbMATH DE number 1098876

zbMATH Keywords

hierarchical parameter estimationprobabilistic Bayesian perspective

Mathematics Subject Classification ID

Point estimation (62F10) Gaussian processes (60G15) Bayesian problems; characterization of Bayes procedures (62C10) Interpolation in approximation theory (41A05) PDEs in connection with statistics (35Q62)

Cites Work

• Weak convergence and empirical processes. With applications to statistics
• An asymptotically optimal window selection rule for kernel density estimates
• Interpolation of spatial data. Some theory for kriging
• Studies in the history of probability and statistics XLIX: On the Matérn correlation family
• Bayesian Inference for Non-Stationary Spatial Covariance Structure via Spatial Deformations
• ON STATIONARY PROCESSES IN THE PLANE
• An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach
• Consistency of cross validation for comparing regression procedures
• Title not available (Why is that?)
• Operator-adapted wavelets, fast solvers, and numerical homogenization. From a game theoretic approach to numerical approximation and algorithm design
• Kernel methods in machine learning
• The Predictive Sample Reuse Method with Applications
• The Relationship between Variable Selection and Data Agumentation and a Method for Prediction
• Scattered Data Approximation
• Bayesian inverse problems with Gaussian priors
• Bayesian nonparametrics
• Bayes procedures for adaptive inference in inverse problems for the white noise model
• A comparison of generalized cross validation and modified maximum likelihood for estimating the parameters of a stochastic process
• Interpolation of spatial data -- a stochastic or a deterministic problem?
• Problems with likelihood estimation of covariance functions of spatial Gaussian processes
• Approximation from Shift-Invariant Subspaces of L 2 (ℝ d )
• Oracle inequalities for multi-fold cross validation
• The cross-validated adaptive epsilon-net estimator
• Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification
• Asymptotic properties of a maximum likelihood estimator with data from a Gaussian process
• Advanced Lectures on Machine Learning
• Title not available (Why is that?)
• Cross-validation estimation of covariance parameters under fixed-domain asymptotics
• Posterior consistency for Gaussian process approximations of Bayesian posterior distributions
• Kernel flows: from learning kernels from data into the abyss
• Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
• Learning dynamical systems from data: a simple cross-validation perspective. I: Parametric kernel flows
• Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency

Cited In (11)

• Learning dynamical systems from data: a simple cross-validation perspective. III: Irregularly-sampled time series
• Do ideas have shape? Idea registration as the continuous limit of artificial neural networks
• Computational graph completion
• Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
• Numerical methods for mean field games based on Gaussian processes and Fourier features
• Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation
• Gaussian process hydrodynamics
• Kernel methods are competitive for operator learning
• Error analysis of kernel/GP methods for nonlinear and parametric PDEs
• Learning ``best kernels from data in Gaussian process regression. With application to aerodynamics
• Solving and learning nonlinear PDEs with Gaussian processes

This page was built for publication: Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4956916)