On asymptotic properties of hyperparameter estimators for kernel-based regularization methods
From MaRDI portal
Publication:1797138
Abstract: The kernel-based regularization method has two core issues: kernel design and hyperparameter estimation. In this paper, we focus on the second issue and study the properties of several hyperparameter estimators including the empirical Bayes (EB) estimator, two Stein's unbiased risk estimators (SURE) and their corresponding Oracle counterparts, with an emphasis on the asymptotic properties of these hyperparameter estimators. To this goal, we first derive and then rewrite the first order optimality conditions of these hyperparameter estimators, leading to several insights on these hyperparameter estimators. Then we show that as the number of data goes to infinity, the two SUREs converge to the best hyperparameter minimizing the corresponding mean square error, respectively, while the more widely used EB estimator converges to another best hyperparameter minimizing the expectation of the EB estimation criterion. This indicates that the two SUREs are asymptotically optimal but the EB estimator is not. Surprisingly, the convergence rate of two SUREs is slower than that of the EB estimator, and moreover, unlike the two SUREs, the EB estimator is independent of the convergence rate of to its limit, where is the regression matrix and is the number of data. A Monte Carlo simulation is provided to demonstrate the theoretical results.
Recommendations
- Some properties of regularized kernel methods
- scientific article; zbMATH DE number 5629717
- Asymptotic performance bounds for the kernel estimate
- On the almost everywhere properties of the kernel regression estimate
- Asymptotic property for a linear regression model of kernel estimation
- Approximate minimization of the regularized expected error over kernel models
- scientific article; zbMATH DE number 6961998
- Asymptotic normality of support vector machine variants and other regularized kernel methods
- scientific article; zbMATH DE number 537369
Cites work
- A new kernel-based approach for linear system identification
- Continuous-Time DC Kernel—A Stable Generalized First-Order Spline Kernel
- Convex vs non-convex estimators for regression and sparse estimation: the mean squared error properties of ARD and GLasso
- scientific article; zbMATH DE number 3443026 (Why is no real title available?)
- scientific article; zbMATH DE number 2107836 (Why is no real title available?)
- Kernel methods in system identification, machine learning and function estimation: a survey
- Kernels for linear time invariant system identification
- Maximum Entropy Kernels for System Identification
- Maximum entropy properties of discrete-time first-order stable spline kernel
- On kernel design for regularized LTI system identification
- On the estimation of transfer functions, regularizations and Gaussian processes-revisited
- On the robustness of the Bayes and Wiener estimators under model uncertainty
- On the stability of reproducing kernel Hilbert spaces of discrete-time impulse responses
- Prediction error identification of linear systems: a nonparametric Gaussian regression approach
- Regularized linear system identification using atomic, nuclear and kernel-based norms: the role of the stability constraint
- Ridge Regression: Biased Estimation for Nonorthogonal Problems
- System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques
- The harmonic analysis of kernel functions
- Tuning complexity in regularized kernel-based regression and linear system identification: the robustness of the marginal likelihood estimator
Cited in
(23)- On kernel design for regularized LTI system identification
- On input design for regularized LTI system identification: power-constrained input
- Regularized LTI system identification in the presence of outliers: a variational EM approach
- Nonparametric identification of Kronecker networks
- On embeddings and inverse embeddings of input design for regularized system identification
- Kernel-based system identification with manifold regularization: a Bayesian perspective
- Adaptive regularised kernel-based identification method for large-scale systems with unknown order
- On convexification of system identification criteria
- On semiseparable kernels and efficient implementation for regularized system identification and function estimation
- The existence and uniqueness of solutions for kernel-based system identification
- A shift in paradigm for system identification
- On the regularization and optimization in quantum detector tomography
- When cannot regularization improve the least squares estimate in the kernel-based regularized system identification
- Kernel-based regularized iterative learning control of repetitive linear time-varying systems
- Sequential stabilizing spline algorithm for linear systems: eigenvalue approximation and polishing
- Two-time scale reinforcement learning and applications to production planning
- On convergence of covariance matrix of empirical Bayes hyper-parameter estimator
- The posterior selection method for hyperparameters in regularized least squares method
- Joint input-output identification of unstable systems with kernel regularization
- Reduced-order identification methods: hierarchical algorithm or variable elimination algorithm
- The convergence of the hierarchical identification algorithm and variable elimination algorithm
- Accelerated proximal gradient method for systems with unknown structure using Volterra series
- On kernel design for regularized Volterra series identification of Wiener-Hammerstein systems
This page was built for publication: On asymptotic properties of hyperparameter estimators for kernel-based regularization methods
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q1797138)
