Kernel methods in system identification, machine learning and function estimation: a survey

DOI10.1016/j.automatica.2014.01.001zbMath1298.93342OpenAlexW2092766760MaRDI QIDQ462325

Gianluigi Pillonetto, Francesco Dinuzzo, Tianshi Chen, Lennart Ljung, De Nicolao, Giuseppe

Publication date: 20 October 2014

Published in: Automatica (Search for Journal in Brave)

Full work available at URL: http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-106518

zbMATH Keywords

Gaussian processesinverse problemsreproducing kernel Hilbert spacesprediction error methodsbias-variance trade-offkernel-based regularizationlinear system identificationmodel complexity selection

Mathematics Subject Classification ID

Learning and adaptive systems in artificial intelligence (68T05) Estimation and detection in stochastic control theory (93E10) Linear systems in control theory (93C05) Identification in stochastic control theory (93E12)

Related Items

Bayesian nonparametric identification of Wiener systems ⋮ Identification of flame transfer functions in the presence of intrinsic thermoacoustic feedback and noise ⋮ Linear system identification using the sequential stabilizing spline algorithm ⋮ Closed-form expressions and nonparametric estimation of COVID-19 infection rate ⋮ Regularized linear system identification using atomic, nuclear and kernel-based norms: the role of the stability constraint ⋮ \(H_\infty\)-norm-based optimization for the identification of gray-box LTI state-space model parameters ⋮ On the stability of reproducing kernel Hilbert spaces of discrete-time impulse responses ⋮ A new kernel-based approach to hybrid system identification ⋮ Addressing the relative degree restriction in nonlinear adaptive observers: a high-gain observer approach ⋮ Metaheuristic optimization-based identification of fractional-order systems under stable distribution noises ⋮ System identification of MISO fractional systems: parameter and differentiation order estimation ⋮ Kernels for Linear Time Invariant System Identification ⋮ Transfer function and transient estimation by Gaussian process regression in the frequency domain ⋮ Informative input design for kernel-based system identification ⋮ On kernel design for regularized LTI system identification ⋮ The generalized cross validation filter ⋮ An empirical Bayes approach to identification of modules in dynamic networks ⋮ Kernel-based system identification with manifold regularization: a Bayesian perspective ⋮ Generalized System Identification with Stable Spline Kernels ⋮ Adaptive regularised kernel-based identification method for large-scale systems with unknown order ⋮ Kernel-based identification of non-causal systems with application to inverse model control ⋮ Leading impulse response identification via the elastic net criterion ⋮ Control-oriented regularization for linear system identification ⋮ MIMO ILC using complex-kernel regression and application to precision SEA robots ⋮ Learning linear modules in a dynamic network using regularized kernel-based methods ⋮ Learning nonlinear state-space models using autoencoders ⋮ Kernel-based methods for Volterra series identification ⋮ On semiseparable kernels and efficient implementation for regularized system identification and function estimation ⋮ Deterministic error bounds for kernel-based learning techniques under bounded noise ⋮ Regularized nonparametric Volterra kernel estimation ⋮ Maximum entropy vector kernels for MIMO system identification ⋮ Data-driven switching modeling for MPC using regression trees and random forests ⋮ A new kernel-based approach to system identification with quantized output data ⋮ A nonparametric kernel-based approach to Hammerstein system identification ⋮ Identification of fractional-order systems with unknown initial values and structure ⋮ High precision variational Bayesian inference of sparse linear networks ⋮ Efficient spatio-temporal Gaussian regression via Kalman filtering ⋮ On the mathematical foundations of stable RKHSs ⋮ Tuning complexity in regularized kernel-based regression and linear system identification: the robustness of the marginal likelihood estimator ⋮ Nonlinear system identification in Sobolev spaces ⋮ A regularised fast recursive algorithm for fraction model identification of nonlinear dynamic systems ⋮ The existence and uniqueness of solutions for kernel-based system identification ⋮ On the regularization and optimization in quantum detector tomography ⋮ Stable spline identification of linear systems under missing data ⋮ When cannot regularization improve the least squares estimate in the kernel-based regularized system identification ⋮ On statistical properties of the estimator of impulse response function ⋮ Kernel-based identification with frequency domain side-information ⋮ An LMI framework for contraction-based nonlinear control design by derivatives of Gaussian process regression ⋮ Topology identification of sparse network: a stochastic variational Bayesian approach ⋮ Kernel-based regularized iterative learning control of repetitive linear time-varying systems ⋮ Sliding-Mode Theory Under Feedback Constraints and the Problem of Epidemic Control ⋮ Data‐driven mixed‐sensitivity control with automated weighting functions selection ⋮ A high-order norm-product regularized multiple kernel learning framework for kernel optimization ⋮ Recursive estimation for sparse Gaussian process regression ⋮ Learning dynamical systems using local stability priors ⋮ Nonlinear system identification for multivariable control via discrete-time Chen-Fliess series ⋮ On the almost sure central limit theorem for ARX processes in adaptive tracking ⋮ Non-Linear Interactions and Exchange Rate Prediction: Empirical Evidence Using Support Vector Regression ⋮ An analysis of the SPARSEVA estimate for the finite sample data case ⋮ Prediction-error identification of LPV systems: a nonparametric Gaussian regression approach ⋮ On input design for regularized LTI system identification: power-constrained input ⋮ Towards efficient maximum likelihood estimation of LPV-SS models ⋮ Two-phase selective decentralization to improve reinforcement learning systems with MDP ⋮ Kernel Absolute Summability Is Sufficient but Not Necessary for RKHS Stability ⋮ An instrumental least squares support vector machine for nonlinear system identification ⋮ Identification of extended Hammerstein systems with hysteresis-type input nonlinearities described by Preisach model ⋮ Subspace-based spectrum estimation in innovation models by mixed norm minimization ⋮ Sparse plus low rank network identification: a nonparametric approach ⋮ Model selection for dynamical systems via sparse regression and information criteria ⋮ Facing undermodelling in sign-perturbed-sums system identification ⋮ Bayesian positive system identification: truncated Gaussian prior and hyperparameter estimation ⋮ The quest for the right kernel in Bayesian impulse response identification: the use of OBFs ⋮ Regularized LTI system identification in the presence of outliers: a variational EM approach ⋮ On convexification of system identification criteria ⋮ System identification using kernel-based regularization: new insights on stability and consistency issues ⋮ Recursive nonlinear-system identification using latent variables ⋮ The harmonic analysis of kernel functions ⋮ On asymptotic properties of hyperparameter estimators for kernel-based regularization methods ⋮ Nonparametric identification of linear dynamic errors-in-variables systems ⋮ Control-based algorithms for high dimensional online learning ⋮ A shift in paradigm for system identification ⋮ Sparse RKHS estimation via globally convex optimization and its application in LPV-IO identification ⋮ FRF estimation using multiple kernel-based regularisation ⋮ Application of regularized Savitzky-Golay filters to identification of time-varying systems ⋮ Revisiting total model errors and model validation ⋮ Distribution-free uncertainty quantification for kernel methods by gradient perturbations ⋮ Sparse identification of nonlinear dynamics for model predictive control in the low-data limit ⋮ Nonlinear system identification via data augmentation ⋮ Nonparametric identification of Kronecker networks ⋮ A Scaled Gradient Projection Method for Bayesian Learning in Dynamical Systems ⋮ Linearly Constrained Linear Quadratic Regulator from the Viewpoint of Kernel Methods ⋮ Bayesian frequentist bounds for machine learning and system identification ⋮ Sparse estimation in linear dynamic networks using the stable spline horseshoe prior ⋮ Recovery of Surfaces and Functions in High Dimensions: Sampling Theory and Links to Neural Networks ⋮ Kernel-based identification of asymptotically stable continuous-time linear dynamical systems ⋮ On embeddings and inverse embeddings of input design for regularized system identification ⋮ An efficient implementation for spatial-temporal Gaussian process regression and its applications ⋮ Maximum entropy properties of discrete-time first-order stable spline kernel ⋮ A robust whiteness test for the identification of discrete-time linear models: use of orthonormal transfer functions ⋮ Robust EM kernel-based methods for linear system identification

Uses Software

• System Identification Toolbox
• glmnet

Cites Work

• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• Unnamed Item
• A reproducing kernel Hilbert space approach to functional linear regression
• Prediction error identification of linear systems: a nonparametric Gaussian regression approach
• On covariance function tests used in system identification
• Kernel methods in machine learning
• Bayesian system identification via Markov chain Monte Carlo techniques
• A new kernel-based approach for linear system identification
• Statistical decision theory and Bayesian analysis. 2nd ed
• The truncated SVD as a method for regularization
• Modeling by shortest data description
• A robust generalized Bayes estimator and confidence region for a multivariate normal mean
• Estimation of the mean of a multivariate normal distribution
• Selecting a minimax estimator of a multivariate normal mean
• Minimax ridge regression estimation
• Minimax estimators of the mean of a multivariate normal distribution
• Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation
• Estimating the dimension of a model
• An overview of robust Bayesian analysis. (With discussion)
• Nonparametric input estimation in physiological systems: Problems, methods, and case studies
• Convex analysis approach to d. c. programming: Theory, algorithms and applications
• Use of PRESS residuals in dynamic system identification
• The convex geometry of linear inverse problems
• Regularization networks and support vector machines
• DC programming: overview.
• Bayesian semiparametric Wiener system identification
• Implementation of algorithms for tuning parameters in regularized least squares problems in system identification
• On the estimation of transfer functions, regularizations and Gaussian processes-revisited
• Bayes and empirical Bayes semi-blind deconvolution using eigenfunctions of a prior covariance
• Identification of MIMO Hammerstein models using least squares support vector machines
• Learning rates of least-square regularized regression
• Some results on Tchebycheffian spline functions and stochastic processes
• The inverse problem of radiography
• A new class of generalized Bayes minimax ridge regression estimators
• Consistent identification of Wiener systems: a machine learning viewpoint
• Learning theory estimates via integral operators and their approximations
• Approximate maximum a posteriori with Gaussian process priors
• On early stopping in gradient descent learning
• On the mathematical foundations of learning
• 10.1162/153244302760185252
• Hankel Matrix Rank Minimization with Applications to System Identification and Realization
• System Identification Via Sparse Multiple Kernel-Based Regularization Using Sequential Convex Optimization Techniques
• Sparse Estimation of Polynomial and Rational Dynamical Models
• Regularization Algorithms for Learning That Are Equivalent to Multilayer Networks
• Black-box identification of mimo transfer functions: Asymptotic properties of prediction error models
• A Unifying View of Wiener and Volterra Theory and Polynomial Kernel Regression
• VECTOR VALUED REPRODUCING KERNEL HILBERT SPACES OF INTEGRABLE FUNCTIONS AND MERCER THEOREM
• An Explicit Description of the Reproducing Kernel Hilbert Spaces of Gaussian RBF Kernels
• Interior-Point Method for Nuclear Norm Approximation with Application to System Identification
• Regression and time series model selection in small samples
• A Technique for the Numerical Solution of Certain Integral Equations of the First Kind
• Generalized Cross-Validation as a Method for Choosing a Good Ridge Parameter
• Quantifying the error in estimated transfer functions with application to model order selection
• Asymptotic properties of the least-squares method for estimating transfer functions and disturbance spectra
• Minimax Adaptive Generalized Ridge Regression Estimators
• Consistent identification of NARX models via regularization networks
• Frequency domain system identification with missing data
• Particle Markov Chain Monte Carlo Methods
• Inverse Problem Theory and Methods for Model Parameter Estimation
• Learning Theory
• The regularizing properties of the adjoint gradient method in ill-posed problems
• The Relationship between Variable Selection and Data Agumentation and a Method for Prediction
• The Minimal Error Conjugate Gradient Method is a Regularization Method
• Partially-Linear Least-Squares Regularized Regression for System Identification
• Stein's Estimation Rule and Its Competitors--An Empirical Bayes Approach
• Kernel based partially linear models and nonlinear identification
• A New Kernel-Based Approach for NonlinearSystem Identification
• Estimating the Mean of a Multivariate Normal Population with General Quadratic Loss Function
• Convex Analysis
• A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines
• Convergence Properties of the Spline Fit
• On the Order of Convergence of Natural Cubic Spline Interpolation
• Some Comments on C P
• Theory of Reproducing Kernels
• Probability, Frequency and Reasonable Expectation
• Robust Statistics
• A new look at the statistical model identification