Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation
From MaRDI portal
Publication:4956916
Abstract: Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Mat'ern-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Mat'ern-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Mat'ern-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.
Recommendations
- Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
- Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency
- Hierarchical modeling with gaussian processes
- Properties of the Bayesian parameter estimation of a regression based on Gaussian processes
- scientific article; zbMATH DE number 1098876
Cites work
- scientific article; zbMATH DE number 1223843 (Why is no real title available?)
- scientific article; zbMATH DE number 740116 (Why is no real title available?)
- A comparison of generalized cross validation and modified maximum likelihood for estimating the parameters of a stochastic process
- Advanced Lectures on Machine Learning
- An asymptotically optimal window selection rule for kernel density estimates
- An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach
- Approximation from Shift-Invariant Subspaces of L 2 (ℝ d )
- Asymptotic properties of a maximum likelihood estimator with data from a Gaussian process
- Bayes procedures for adaptive inference in inverse problems for the white noise model
- Bayesian Inference for Non-Stationary Spatial Covariance Structure via Spatial Deformations
- Bayesian inverse problems with Gaussian priors
- Bayesian nonparametrics
- Consistency of cross validation for comparing regression procedures
- Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
- Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification
- Cross-validation estimation of covariance parameters under fixed-domain asymptotics
- Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency
- Interpolation of spatial data -- a stochastic or a deterministic problem?
- Interpolation of spatial data. Some theory for kriging
- Kernel flows: from learning kernels from data into the abyss
- Kernel methods in machine learning
- Learning dynamical systems from data: a simple cross-validation perspective. I: Parametric kernel flows
- ON STATIONARY PROCESSES IN THE PLANE
- Operator-adapted wavelets, fast solvers, and numerical homogenization. From a game theoretic approach to numerical approximation and algorithm design
- Oracle inequalities for multi-fold cross validation
- Posterior consistency for Gaussian process approximations of Bayesian posterior distributions
- Problems with likelihood estimation of covariance functions of spatial Gaussian processes
- Scattered Data Approximation
- Studies in the history of probability and statistics XLIX: On the Matérn correlation family
- The Predictive Sample Reuse Method with Applications
- The Relationship between Variable Selection and Data Agumentation and a Method for Prediction
- The cross-validated adaptive epsilon-net estimator
- Weak convergence and empirical processes. With applications to statistics
Cited in
(11)- Learning dynamical systems from data: a simple cross-validation perspective. III: Irregularly-sampled time series
- Do ideas have shape? Idea registration as the continuous limit of artificial neural networks
- Computational graph completion
- Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
- Numerical methods for mean field games based on Gaussian processes and Fourier features
- Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation
- Gaussian process hydrodynamics
- Kernel methods are competitive for operator learning
- Error analysis of kernel/GP methods for nonlinear and parametric PDEs
- Learning ``best kernels from data in Gaussian process regression. With application to aerodynamics
- Solving and learning nonlinear PDEs with Gaussian processes
This page was built for publication: Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4956916)
