Fast learning rates in statistical inference through aggregation
From MaRDI portal
Publication:2388975
Abstract: We develop minimax optimal risk bounds for the general learning task consisting in predicting as well as the best function in a reference set up to the smallest possible additive term, called the convergence rate. When the reference set is finite and when denotes the size of the training data, we provide minimax convergence rates of the form with tight evaluation of the positive constant and with exact , the latter value depending on the convexity of the loss function and on the level of noise in the output distribution. The risk upper bounds are based on a sequential randomized algorithm, which at each step concentrates on functions having both low risk and low variance with respect to the previous step prediction function. Our analysis puts forward the links between the probabilistic and worst-case viewpoints, and allows to obtain risk bounds unachievable with the standard statistical learning approach. One of the key ideas of this work is to use probabilistic inequalities with respect to appropriate (Gibbs) distributions on the prediction function space instead of using them with respect to the distribution generating the data. The risk lower bounds are based on refinements of the Assouad lemma taking particularly into account the properties of the loss function. Our key example to illustrate the upper and lower bounds is to consider the -regression setting for which an exhaustive analysis of the convergence rates is given while ranges in .
Recommendations
- Fast rates in statistical and online learning
- On Statistical Efficiency in Learning
- Fast learning from \(\alpha\)-mixing observations
- Optimal rates of aggregation in classification under low noise assumption
- Rates of convergence in active learning
- The speed of sequential asymptotic learning
- scientific article; zbMATH DE number 4031026
- Probability and plurality for aggregations of learning machines
- Fast Convergence Rates for Distributed Non-Bayesian Learning
Cites work
- A game of prediction with expert advice
- A New Lower Bound for Multiple Hypothesis Testing
- Aggregated estimators and empirical complexity for least square regression
- Any Discrimination Rule Can Have an Arbitrarily Bad Probability of Error for Finite Sample Size
- Approximation dans les espaces m�triques et th�orie de l'estimation
- Central limit theorems for empirical measures
- Combinatorial methods in density estimation
- Combining different procedures for adaptive regression
- Estimation des densit�s: risque minimax
- Estimation of dependences based on empirical data. Transl. from the Russian by Samuel Kotz
- Fast learning rates for plug-in classifiers
- Fast learning rates in statistical inference through aggregation
- How to better use expert advice
- How to use expert advice
- scientific article; zbMATH DE number 2089353 (Why is no real title available?)
- scientific article; zbMATH DE number 3907540 (Why is no real title available?)
- scientific article; zbMATH DE number 1344902 (Why is no real title available?)
- scientific article; zbMATH DE number 3446442 (Why is no real title available?)
- scientific article; zbMATH DE number 823069 (Why is no real title available?)
- scientific article; zbMATH DE number 893887 (Why is no real title available?)
- scientific article; zbMATH DE number 1405931 (Why is no real title available?)
- scientific article; zbMATH DE number 3252891 (Why is no real title available?)
- Information-theoretic determination of minimax rates of convergence
- Information-theoretic upper and lower bounds for statistical estimation
- Introduction to nonparametric estimation
- Iterative feature selection in least square regression estimation
- Learning by mirror averaging
- Learning Theory
- Learning Theory
- Learning Theory and Kernel Machines
- Local Rademacher complexities and oracle inequalities in risk minimization. (2004 IMS Medallion Lecture). (With discussions and rejoinder)
- Lower bounds for the rate of convergence in nonparametric pattern recognition
- Measuring mass concentrations and estimating density contour clusters -- An excess mass approach
- On prediction of individual sequences
- On the Generalization Ability of On-Line Learning Algorithms
- Optimal aggregation of classifiers in statistical learning.
- Optimal rates of aggregation in classification under low noise assumption
- Prediction, Learning, and Games
- Sequential prediction of individual sequences under general loss functions
- Sequential Procedures for Aggregating Arbitrary Estimators of a Conditional Mean
- Simultaneous adaptation to the margin and to complexity in classification
- Smooth discrimination analysis
- Some applications of concentration inequalities to statistics
- Statistical learning theory and stochastic optimization. Ecole d'Eté de Probabilitiés de Saint-Flour XXXI -- 2001.
- Suboptimality of Penalized Empirical Risk Minimization in Classification
- The progressive mixture estimator for regression trees.
- Theory of Classification: a Survey of Some Recent Advances
- Universal prediction
Cited in
(32)- Simpler PAC-Bayesian bounds for hostile data
- A quasi-Bayesian perspective to online clustering
- On the exponentially weighted aggregate with the Laplace prior
- On the optimality of the empirical risk minimization procedure for the convex aggregation problem
- PAC-Bayesian estimation and prediction in sparse additive models
- Upper bounds and aggregation in bipartite ranking
- On the optimality of the aggregate with exponential weights for low temperatures
- Suboptimality of constrained least squares and improvements via non-linear predictors
- Distribution-free robust linear regression
- Exponential weights in multivariate regression and a low-rankness favoring prior
- Exact lower bounds for the agnostic probably-approximately-correct (PAC) machine learning model
- Fast rates by transferring from auxiliary hypotheses
- On aggregation for heavy-tailed classes
- Fast learning rates in statistical inference through aggregation
- Empirical risk minimization is optimal for the convex aggregation problem
- Optimal learning with \textit{Q}-aggregation
- Sparse regression learning by aggregation and Langevin Monte-Carlo
- Mirror averaging with sparsity priors
- Active nearest-neighbor learning in metric spaces
- Optimal quantum sample complexity of learning algorithms
- Fast rates for general unbounded loss functions: from ERM to generalized Bayes
- scientific article; zbMATH DE number 7625184 (Why is no real title available?)
- Empirical entropy, minimax regret and minimax risk
- Prediction of time series by statistical learning: general losses and fast rates
- Fast rates in statistical and online learning
- Variance function estimation in regression model via aggregation procedures
- User-friendly Introduction to PAC-Bayes Bounds
- Orthogonal statistical learning
- Relaxing the i.i.d. assumption: adaptively minimax optimal regret via root-entropic regularization
- A tight excess risk bound via a unified PAC-Bayesian-Rademacher-Shtarkov-MDL complexity
- Optimal learning with Bernstein online aggregation
- Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processes
This page was built for publication: Fast learning rates in statistical inference through aggregation
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2388975)
