Iterative feature selection in least square regression estimation

DOI10.1214/07-AIHP106MaRDI QIDQ731450zbMATH OpenOpenAlexFDO

Authors Pierre Alquier

Publication date 7 October 2009

Published in Annales de l'Institut Henri Poincaré. Probabilités et Statistiques (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/math/0511299, https://eudml.org/doc/77964

zbMATH Keywords

confidence regionsstatistical learningsupport vector machinesregression estimationthresholding methods

Mathematics Subject Classification ID

Nonparametric regression and quantile regression (62G08) Learning and adaptive systems in artificial intelligence (68T05) Nonparametric tolerance and confidence regions (62G15)

Abstract: In this paper, we focus on regression estimation in both the inductive and the transductive case. We assume that we are given a set of features (which can be a base of functions, but not necessarily). We begin by giving a deviation inequality on the risk of an estimator in every model defined by using a single feature. These models are too simple to be useful by themselves, but we then show how this result motivates an iterative algorithm that performs feature selection in order to build a suitable estimator. We prove that every selected feature actually improves the performance of the estimator. We give all the estimators and results at first in the inductive case, which requires the knowledge of the distribution of the design, and then in the transductive case, in which we do not need to know this distribution.

Recommendations

• Lasso, iterative feature selection and the correlation selector: oracle inequalities and numerical performances
• An algorithm for iterative selection of blocks of features
• Least angle regression. (With discussion)
• Model selection for regularized least-squares algorithm in learning theory
• Least square regression with \(l^{p}\)-coefficient regularization

Cites work

• scientific article; zbMATH DE number 1332320 (Why is no real title available?)
• scientific article; zbMATH DE number 1522808 (Why is no real title available?)
• Aggregated estimators and empirical complexity for least square regression
• Aggregating regression procedures to improve performance
• An introduction to support vector machines and other kernel-based learning methods.
• Approximation and learning by greedy algorithms
• Boosting the margin: a new explanation for the effectiveness of voting methods
• Density estimation by wavelet thresholding
• Ideal spatial adaptation by wavelet shrinkage
• Optimal aggregation of classifiers in statistical learning.
• PAC-Bayesian Generalisation Error Bounds for Gaussian Process Classification
• Recursive aggregation of estimators by the mirror descent algorithm with averaging
• Regression in random design and warped wavelets
• Statistical learning theory and stochastic optimization. Ecole d'Eté de Probabilitiés de Saint-Flour XXXI -- 2001.
• Symmetrization approach to concentration inequalities for empirical processes.
• Wavelets, approximation, and statistical applications

Cited in

(8)

• Iterative selection using orthogonal regression techniques
• Tight conditions for consistency of variable selection in the context of high dimensionality
• scientific article; zbMATH DE number 1708089 (Why is no real title available?)
• An algorithm for iterative selection of blocks of features
• Fast learning rates in statistical inference through aggregation
• scientific article; zbMATH DE number 1931847 (Why is no real title available?)
• Lasso, iterative feature selection and the correlation selector: oracle inequalities and numerical performances
• Density estimation with quadratic loss: a confidence intervals method

This page was built for publication: Iterative feature selection in least square regression estimation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q731450)