Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation
From MaRDI portal
Abstract: Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Mat'ern-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Mat'ern-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Mat'ern-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.
Recommendations
- Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
- Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency
- Hierarchical modeling with gaussian processes
- Properties of the Bayesian parameter estimation of a regression based on Gaussian processes
- scientific article; zbMATH DE number 1098876
Cites work
- A comparison of generalized cross validation and modified maximum likelihood for estimating the parameters of a stochastic process
- Advanced Lectures on Machine Learning
- An asymptotically optimal window selection rule for kernel density estimates
- An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach
- Approximation from Shift-Invariant Subspaces of L 2 (ℝ d )
- Asymptotic properties of a maximum likelihood estimator with data from a Gaussian process
- Bayes procedures for adaptive inference in inverse problems for the white noise model
- Bayesian Inference for Non-Stationary Spatial Covariance Structure via Spatial Deformations
- Bayesian inverse problems with Gaussian priors
- Bayesian nonparametrics
- Consistency of cross validation for comparing regression procedures
- Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
- Cross validation and maximum likelihood estimations of hyper-parameters of Gaussian processes with model misspecification
- Cross-validation estimation of covariance parameters under fixed-domain asymptotics
- scientific article; zbMATH DE number 1223843 (Why is no real title available?)
- scientific article; zbMATH DE number 740116 (Why is no real title available?)
- Hyperparameter estimation in Bayesian MAP estimation: parameterizations and consistency
- Interpolation of spatial data -- a stochastic or a deterministic problem?
- Interpolation of spatial data. Some theory for kriging
- Kernel flows: from learning kernels from data into the abyss
- Kernel methods in machine learning
- Learning dynamical systems from data: a simple cross-validation perspective. I: Parametric kernel flows
- ON STATIONARY PROCESSES IN THE PLANE
- Operator-adapted wavelets, fast solvers, and numerical homogenization. From a game theoretic approach to numerical approximation and algorithm design
- Oracle inequalities for multi-fold cross validation
- Posterior consistency for Gaussian process approximations of Bayesian posterior distributions
- Problems with likelihood estimation of covariance functions of spatial Gaussian processes
- Scattered Data Approximation
- Studies in the history of probability and statistics XLIX: On the Matérn correlation family
- The cross-validated adaptive epsilon-net estimator
- The Predictive Sample Reuse Method with Applications
- The Relationship between Variable Selection and Data Agumentation and a Method for Prediction
- Weak convergence and empirical processes. With applications to statistics
Cited in
(18)- Learning ``best kernels from data in Gaussian process regression. With application to aerodynamics
- Do ideas have shape? Idea registration as the continuous limit of artificial neural networks
- Solving and learning nonlinear PDEs with Gaussian processes
- Numerical methods for mean field games based on Gaussian processes and Fourier features
- Computational graph completion
- Learning dynamical systems from data: a simple cross-validation perspective. III: Irregularly-sampled time series
- Convergence of Gaussian process regression with estimated hyper-parameters and applications in Bayesian inverse problems
- Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation
- Gaussian process hydrodynamics
- Kernel methods are competitive for operator learning
- Error analysis of kernel/GP methods for nonlinear and parametric PDEs
- Smoothness estimation for Whittle-Matérn processes on closed Riemannian manifolds
- Comparing scale parameter estimators for Gaussian process interpolation with the Brownian motion prior: leave-one-out cross validation and maximum likelihood
- An asymptotic study of the joint maximum likelihood estimation of the regularity and the amplitude parameters of a periodized Matérn model
- Hyperparameter optimization for randomized algorithms: a case study on random features
- Sparse Cholesky factorization for solving nonlinear PDEs via Gaussian processes
- Gaussian measures conditioned on nonlinear observations: consistency, MAP estimators, and simulation
- Solving functional PDEs with Gaussian processes and applications to functional renormalization group equations
This page was built for publication: Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q4956916)
