Aggregation using input-output trade-off
From MaRDI portal
Publication:2317264
Abstract: In this paper, we introduce a new learning strategy based on a seminal idea of Mojirsheibani (1999, 2000, 2002a, 2002b), who proposed a smart method for combining several classifiers, relying on a consensus notion. In many aggregation methods, the prediction for a new observation x is computed by building a linear or convex combination over a collection of basic estimators r1(x),. .. , rm(x) previously calibrated using a training data set. Mojirsheibani proposes to compute the prediction associated to a new observation by combining selected outputs of the training examples. The output of a training example is selected if some kind of consensus is observed: the predictions computed for the training example with the different machines have to be "similar" to the prediction for the new observation. This approach has been recently extended to the context of regression in Biau et al. (2016). In the original scheme, the agreement condition is actually required to hold for all individual estimators, which appears inadequate if there is one bad initial estimator. In practice, a few disagreements are allowed ; for establishing the theoretical results, the proportion of estimators satisfying the condition is required to tend to 1. In this paper, we propose an alternative procedure, mixing the previous consensus ideas on the predictions with the Euclidean distance computed between entries. This may be seen as an alternative approach allowing to reduce the effect of a possibly bad estimator in the initial list, using a constraint on the inputs. We prove the consistency of our strategy in classification and in regression. We also provide some numerical experiments on simulated and real data to illustrate the benefits of this new aggregation method. On the whole, our practical study shows that our method may perform much better than the original combination technique, and, in particular, exhibit far less variance. We also show on simulated examples that this procedure mixing inputs and outputs is still robust to high dimensional inputs.
Recommendations
Cites work
- A COMPARISON STUDY OF SOME COMBINED CLASSIFIERS
- A distribution-free theory of nonparametric regression
- A kernel-based combined classification rule
- A nonlinear aggregation type classifier
- Adaptive Regression by Mixing
- Aggregated estimators and empirical complexity for least square regression
- Aggregating regression procedures to improve performance
- Aggregation and Sparsity Via ℓ1 Penalized Least Squares
- Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
- Aggregation for Gaussian regression
- An almost surely optimal combined classification rule
- An equivalence theorem for \(L_ 1\) convergence of the kernel regression estimate
- COBRA: a combined regression strategy
- Combining Classifiers via Discretization
- Combining different procedures for adaptive regression
- Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII -- 2003.
- Functional aggregation for nonparametric regression.
- scientific article; zbMATH DE number 1522808 (Why is no real title available?)
- scientific article; zbMATH DE number 893887 (Why is no real title available?)
- Model selection in nonparametric regression
- Sparsity oracle inequalities for the Lasso
- Statistical learning theory and stochastic optimization. Ecole d'Eté de Probabilitiés de Saint-Flour XXXI -- 2001.
Cited in
(5)- Problem of aggregating of an input-output model and duality
- KFC: A clusterwise supervised learning procedure based on the aggregation of distances
- A nearest-neighbor-based ensemble classifier and its large-sample optimality
- Extending the Macsum aggregation to interval-valued inputs
- Some results on statistics and functional data analysis
This page was built for publication: Aggregation using input-output trade-off
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2317264)
