Distribution-free uncertainty quantification for kernel methods by gradient perturbations

DOI10.1007/S10994-019-05822-1zbMATH Open1494.62012arXiv1812.09632OpenAlexW3100536890WikidataQ127610749 ScholiaQ127610749MaRDI QIDQ2320596FDOQ2320596

Krisztián B. Kis, Balázs Csanád Csáji

Publication date: 23 August 2019

Published in: Machine Learning (Search for Journal in Brave)

Abstract: We propose a data-driven approach to quantify the uncertainty of models constructed by kernel methods. Our approach minimizes the needed distributional assumptions, hence, instead of working with, for example, Gaussian processes or exponential families, it only requires knowledge about some mild regularity of the measurement noise, such as it is being symmetric or exchangeable. We show, by building on recent results from finite-sample system identification, that by perturbing the residuals in the gradient of the objective function, information can be extracted about the amount of uncertainty our model has. Particularly, we provide an algorithm to build exact, non-asymptotically guaranteed, distribution-free confidence regions for ideal, noise-free representations of the function we try to estimate. For the typical convex quadratic problems and symmetric noises, the regions are star convex centered around a given nominal estimate, and have efficient ellipsoidal outer approximations. Finally, we illustrate the ideas on typical kernel methods, such as LS-SVC, KRR, $varepsilon$-SVR and kernelized LASSO.

Full work available at URL: https://arxiv.org/abs/1812.09632

Recommendations

• Probabilistic kernel support vector machines
• Efficient methods for robust classification under uncertainty in kernel matrices
• Toward a Kernel-Based Uncertainty Decomposition Framework for Data and Models
• Maximum likelihood estimation and uncertainty quantification for Gaussian process approximation of deterministic functions
• Can we trust Bayesian uncertainty quantification from Gaussian process priors with squared exponential covariance kernel?

zbMATH Keywords

classificationnonparametric regressionconfidence regionskernel methodssupport vector machinesdistribution-free methods

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Classification and discrimination; cluster analysis (statistical aspects) (62H30) Learning and adaptive systems in artificial intelligence (68T05) Nonparametric tolerance and confidence regions (62G15)

Cites Work

• Title not available (Why is that?)
• Title not available (Why is that?)
• Theory of Reproducing Kernels
• Honest confidence regions for nonparametric regression
• Some results on Tchebycheffian spline functions and stochastic processes
• An introduction to support vector machines and other kernel-based learning methods.
• Mathematical Foundations of Infinite-Dimensional Statistical Models
• Support Vector Machines
• Kernel methods in machine learning
• Title not available (Why is that?)
• Permutation, parametric and bootstrap tests of hypotheses.
• Title not available (Why is that?)
• Kernel methods in system identification, machine learning and function estimation: a survey
• Title not available (Why is that?)
• Nonparametric regression, confidence regions and regularization
• Guaranteed non-asymptotic confidence regions in system identification
• Sign-Perturbed Sums: A New System Identification Approach for Constructing Exact Non-Asymptotic Confidence Regions in Linear Regression Models

Cited In (5)

• Bayesian frequentist bounds for machine learning and system identification
• Facing undermodelling in sign-perturbed-sums system identification
• A simple condition for the boundedness of sign-perturbed-sums (SPS) confidence regions
• Toward a Kernel-Based Uncertainty Decomposition Framework for Data and Models
• Scalable bounding of predictive uncertainty in regression problems with SLAC

This page was built for publication: Distribution-free uncertainty quantification for kernel methods by gradient perturbations

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2320596)