Boosting as a kernel-based method

DOI10.1007/S10994-019-05797-ZMaRDI QIDQ2331677zbMATH OpenOpenAlexFDO

Authors Aleksandr Y. Aravkin, Giulio Bottegal, Gianluigi Pillonetto

Publication date 30 October 2019

Published in Machine Learning (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1608.02485

zbMATH Keywords

boostingkernel-based methodsreproducing kernel Hilbert spacesrobust estimationweak learners

Mathematics Subject Classification ID

Classification and discrimination; cluster analysis (statistical aspects) (62H30) Learning and adaptive systems in artificial intelligence (68T05)

Abstract: Boosting combines weak (biased) learners to obtain effective learning algorithms for classification and prediction. In this paper, we show a connection between boosting and kernel-based methods, highlighting both theoretical and practical applications. In the context of $el{l}_2$ boosting, we start with a weak linear learner defined by a kernel $K$. We show that boosting with this learner is equivalent to estimation with a special {it boosting kernel} that depends on $K$, as well as on the regression matrix, noise variance, and hyperparameters. The number of boosting iterations is modeled as a continuous hyperparameter, and fit along with other parameters using standard techniques. We then generalize the boosting kernel to a broad new class of boosting approaches for more general weak learners, including those based on the $el{l}_1$, hinge and Vapnik losses. The approach allows fast hyperparameter tuning for this general class, and has a wide range of applications, including robust regression and classification. We illustrate some of these applications with numerical examples on synthetic and real data.

Recommendations

• Boosting as a regularized path to a maximum margin classifier
• scientific article; zbMATH DE number 1950578
• \(L_{2}\) boosting in kernel regression
• On boosting kernel regression
• Boosting. Foundations and algorithms.

Cites work

• A decision-theoretic generalization of on-line learning and an application to boosting
• A new kernel-based approach for linear system identification
• Additive logistic regression: a statistical view of boosting. (With discussion and a rejoinder by the authors)
• Arcing classifiers. (With discussion)
• Boosting algorithms: regularization, prediction and model fitting
• Boosting ridge regression
• Boosting With theL₂Loss
• Boosting. Foundations and algorithms.
• Efficient margin maximizing with boosting
• Elastic-net regularization in learning theory
• Generalized Kalman smoothing: modeling and algorithms
• scientific article; zbMATH DE number 1804115 (Why is no real title available?)
• scientific article; zbMATH DE number 44406 (Why is no real title available?)
• scientific article; zbMATH DE number 45848 (Why is no real title available?)
• scientific article; zbMATH DE number 1332320 (Why is no real title available?)
• Learning rates of least-square regularized regression
• Learning theory estimates via integral operators and their approximations
• Mercer theorem for RKHS on noncompact sets
• Model Selection and the Principle of Minimum Description Length
• Multi-class AdaBoost
• On the Eigenvalues of a Class of Integral Equations Arising in Laser Theory
• On the influence of the kernel on the consistency of support vector machines
• Quantile regression.
• Reappraising Medfly Longevity
• Regularization and Variable Selection Via the Elastic Net
• Regularization networks and support vector machines
• Regularized simultaneous model selection in multiple quantiles regression
• Robust EM kernel-based methods for linear system identification
• Robust Statistics
• Sequential greedy approximation for certain convex optimization problems
• Smoothing Parameter Selection in Nonparametric Regression Using an Improved Akaike Information Criterion
• Sparse regression and support recovery with \(\mathbb{L}_2\)-boosting algorithms
• The Bayesian elastic net
• The boosting approach to machine learning: an overview
• The elements of statistical learning. Data mining, inference, and prediction
• Weak greedy algorithms

Cited in

(9)

• On boosting kernel regression
• scientific article; zbMATH DE number 1717314 (Why is no real title available?)
• Boosting as a regularized path to a maximum margin classifier
• Fully corrective boosting with arbitrary loss and regularization
• Boosted kernel ridge regression: optimal learning rates and early stopping
• Algorithmic Learning Theory
• \(L_{2}\) boosting in kernel regression
• Fully corrective gradient boosting with squared hinge: fast learning rates and early stopping
• On boosting kernel density methods for multivariate data: density estimation and classification

Describes a project that uses

Uses Software

• ElemStatLearn
• AdaBoost.RT
• AdaBoost.MH
• AdaBoost-SAMME

This page was built for publication: Boosting as a kernel-based method

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2331677)