Exploiting Hessian matrix and trust-region algorithm in hyperparameters estimation of Gaussian process (Q2491022)

Nonparametric Bayesian approach to Gaussian regression is considered. The training data \((x_i,t_i)_{i=1}^N\) consist of output \(t_i\in\mathbb{R}\) and input vectors \(x_i\in \mathbb{R}^L\). Their distribution is described by \[ \mathbf{P}(t| \;x,\Theta)\propto \exp\left( -{1\over 2}t^TC^{-1}(\vartheta)t \right), \] where \(C\) is a matrix with entries \[ c(x_i,x_j,\Theta)= \alpha\exp \left( -{1\over 2}\sum_{l=1}^L (x_i^{(l)}-x_j^{(l)})^2d_l \right) +\nu\delta_{ij}, \] \(\Theta=(\alpha,d_1,\dots,d_L,\nu)\) being the hyperparameter. It is proposed to estimate \(\Theta\) using maximum likelihood, i.e. to minimize the negative log-likelihood \(L(\Theta)={1\over 2}\log\det C(\Theta)+{1\over 2}t^TC^{-1}(\Theta)t\). The authors describe the form of the Hessian matrix for \(L\) and propose a second-order trust-region algorithm for the minimization of \(L\). Results of numerical simulations are presented.