Consistency and robustness of kernel-based regression in convex risk minimization

DOI10.3150/07-BEJ5102zbMATH Open1129.62031arXiv0709.0626WikidataQ59196404 ScholiaQ59196404MaRDI QIDQ2469652FDOQ2469652

Authors: Andreas Christmann, Ingo Steinwart 

Publication date: 6 February 2008

Published in: Bernoulli (Search for Journal in Brave)

Abstract: We investigate statistical properties for a broad class of modern kernel-based regression (KBR) methods. These kernel methods were developed during the last decade and are inspired by convex risk minimization in infinite-dimensional Hilbert spaces. One leading example is support vector regression. We first describe the relationship between the loss function $L$ of the KBR method and the tail of the response variable. We then establish the $L$-risk consistency for KBR which gives the mathematical justification for the statement that these methods are able to ``learn. Then we consider robustness properties of such kernel methods. In particular, our results allow us to choose the loss function and the kernel to obtain computationally tractable and consistent KBR methods that have bounded influence functions. Furthermore, bounds for the bias and for the sensitivity curve, which is a finite sample version of the influence function, are developed, and the relationship between KBR and classical $M$ estimators is discussed.

Full work available at URL: https://arxiv.org/abs/0709.0626

Recommendations

• On robustness properties of convex risk minimization methods for pattern recognition
• Robustness of reweighted least squares kernel based regression
• Bouligand derivatives and robustness of support vector machines for regression
• On the robustness of regularized pairwise learning methods based on kernels
• Robust regression function estimation

zbMATH Keywords

consistencyinfluence functionsupport vector regressionsensitivity curve

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Applications of functional analysis in probability theory and statistics (46N30) Numerical methods for mathematical programming, optimization and variational techniques (65K99)

Cites Work

• Title not available (Why is that?)
• Robust Statistics
• Least Median of Squares Regression
• Title not available (Why is that?)
• Title not available (Why is that?)
• Real Analysis and Probability
• On robustness properties of convex risk minimization methods for pattern recognition
• The Influence Curve and Its Role in Robust Estimation
• Title not available (Why is that?)
• Title not available (Why is that?)
• A distribution-free theory of nonparametric regression
• On the influence of the kernel on the consistency of support vector machines
• Title not available (Why is that?)
• Convex functions, monotone operators and differentiability.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Sums of independent Banach space valued random variables
• Consistency of Support Vector Machines and Other Regularized Kernel Classifiers
• Title not available (Why is that?)
• Some properties of regularized kernel methods
• Title not available (Why is that?)
• 10.1162/1532443041827925
• Function Classes That Approximate the Bayes Risk
• An approach to model complex high-dimensional insurance data
• How to compare different loss functions and their risks
• Title not available (Why is that?)
• Aspects of robust linear regression
• Consistency of kernel-based quantile regression

Cited In (45)

• Deep learning theory of distribution regression with CNNs
• Robust learning from bites for data mining
• Error analysis of the kernel regularized regression based on refined convex losses and RKBSs
• Distributed robust regression with correntropy losses and regularization kernel networks
• On robustness properties of convex risk minimization methods for pattern recognition
• Performance analysis of the LapRSSLG algorithm in learning theory
• Estimation of the bandwidth parameter in Nadaraya-Watson kernel non-parametric regression based on universal threshold level
• Error analysis on Hérmite learning with gradient data
• Privacy-Preserving Parametric Inference: A Case for Robust Statistics
• Learning with Convex Loss and Indefinite Kernels
• Robust kernel-based distribution regression
• Analysis of support vector machines regression
• Learning rates for the kernel regularized regression with a differentiable strongly convex loss
• Detecting influential observations in kernel PCA
• Robust nonparametric kernel regression estimator
• A two-experiment approach to Wiener system identification
• On the robustness of regularized pairwise learning methods based on kernels
• Convergence analysis for kernel-regularized online regression associated with an RRKHS
• Prediction of dynamical time series using kernel based regression and smooth splines
• Robustness of reweighted least squares kernel based regression
• On qualitative robustness of support vector machines
• Bouligand derivatives and robustness of support vector machines for regression
• Fast rates of minimum error entropy with heavy-tailed noise
• Asymptotic normality of support vector machine variants and other regularized kernel methods
• A statistical learning assessment of Huber regression
• The performance of semi-supervised Laplacian regularized regression with the least square loss
• Loan pricing under estimation risk
• Optimality of robust online learning
• Distribution-free consistency of empirical risk minimization and support vector regression
• Consistency of support vector machines for forecasting the evolution of an unknown ergodic dynamical system from observations with unknown noise
• Adaptive kernel methods using the balancing principle
• Robustness by reweighting for kernel estimators: an overview
• Trading Variance Reduction with Unbiasedness: The Regularized Subspace Information Criterion for Robust Model Selection in Kernel Regression
• Identifying outliers using multiple kernel canonical correlation analysis with application to imaging genetics
• Kernel-Based Partial Permutation Test for Detecting Heterogeneous Functional Relationship
• On a strategy to develop robust and simple tariffs from motor vehicle insurance data
• Feasible generalized least squares using support vector regression
• A review on consistency and robustness properties of support vector machines for heavy-tailed distributions
• Testing subspace restrictions in the presence of high dimensional nuisance parameters
• Frameworks and results in distributionally robust optimization
• An Invariance Property of Predictors in Kernel-Induced Hypothesis Spaces
• Title not available (Why is that?)
• Robust pairwise learning with Huber loss
• Modification of the adaptive Nadaraya-Watson kernel method for nonparametric regression (simulation study)
• Learning from dependent observations

This page was built for publication: Consistency and robustness of kernel-based regression in convex risk minimization

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2469652)