Deviation optimal learning using greedy Q-aggregation

DOI10.1214/12-AOS1025zbMATH Open1257.62037MaRDI QIDQ693750FDOQ693750

Authors Dong Dai, Philippe Rigollet, Tong Zhang

Publication date 10 December 2012

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1203.2507

zbMATH Keywords

model selectionregressionmodel averagingoracle inequalitiesgreedy algorithmlower boundsdeviation boundsexponential weightsdeviation suboptimality

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Nonparametric regression and quantile regression (62G08) Asymptotic properties of nonparametric inference (62G20) Methods of reduced gradient type (90C52)

Abstract: Given a finite family of functions, the goal of model selection aggregation is to construct a procedure that mimics the function from this family that is the closest to an unknown regression function. More precisely, we consider a general regression model with fixed design and measure the distance between functions by the mean squared error at the design points. While procedures based on exponential weights are known to solve the problem of model selection aggregation in expectation, they are, surprisingly, sub-optimal in deviation. We propose a new formulation called Q-aggregation that addresses this limitation; namely, its solution leads to sharp oracle inequalities that are optimal in a minimax sense. Moreover, based on the new formulation, we design greedy Q-aggregation procedures that produce sparse aggregation models achieving the optimal rate. The convergence and performance of these greedy procedures are illustrated and compared with other standard methods on simulated examples.

Recommendations

• Optimal learning with \textit{Q}-aggregation
• Hyper-sparse optimal aggregation
• Learning Theory and Kernel Machines
• Aggregation for Gaussian regression
• Optimal exponential bounds for aggregation of density estimators

Cites work

• scientific article; zbMATH DE number 5764862 (Why is no real title available?)
• scientific article; zbMATH DE number 1321826 (Why is no real title available?)
• scientific article; zbMATH DE number 1522808 (Why is no real title available?)
• scientific article; zbMATH DE number 2107836 (Why is no real title available?)
• scientific article; zbMATH DE number 6176343 (Why is no real title available?)
• scientific article; zbMATH DE number 1405931 (Why is no real title available?)
• A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
• A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training
• Adaptive estimation of a quadratic functional by model selection.
• Aggregated estimators and empirical complexity for least square regression
• Aggregation by Exponential Weighting and Sharp Oracle Inequalities
• Aggregation via empirical risk minimization
• Exponential screening and optimal rates of sparse estimation
• Functional aggregation for nonparametric regression.
• Hyper-sparse optimal aggregation
• Kullback-Leibler aggregation and misspecified generalized linear models
• Learning Theory and Kernel Machines
• Learning by mirror averaging
• On the optimality of the aggregate with exponential weights for low temperatures
• Sharp oracle inequalities for aggregation of affine estimators
• Trading accuracy for sparsity in optimization problems with sparsity constraints
• Universal approximation bounds for superpositions of a sigmoidal function

Cited in

(22)

• Solution of linear ill-posed problems by model selection and aggregation
• Model aggregation for doubly divided data with large size and large dimension
• Aggregation of affine estimators
• Histopathological imaging‐based cancer heterogeneity analysis via penalized fusion with model averaging
• Hyper-sparse optimal aggregation
• Martingale-residual-based greedy model averaging for high-dimensional current status data
• Rank-Based Greedy Model Averaging for High-Dimensional Survival Data
• Statistical inference in compound functional models
• scientific article; zbMATH DE number 7306896 (Why is no real title available?)
• Estimation and Inference for High-Dimensional Generalized Linear Models with Knowledge Transfer
• Second-order Stein: SURE for SURE and other applications in high-dimensional inference
• An adaptive multiclass nearest neighbor classifier
• Sharp oracle inequalities for aggregation of affine estimators
• Optimal and Safe Estimation for High-Dimensional Semi-Supervised Learning
• Localized Gaussian width of \(M\)-convex hulls with applications to Lasso and convex aggregation
• User-friendly Introduction to PAC-Bayes Bounds
• Optimal Kullback-Leibler aggregation in mixture density estimation by maximum likelihood
• Optimal learning with \textit{Q}-aggregation
• PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting
• Optimal bounds for aggregation of affine estimators
• Optimal learning with Bernstein online aggregation
• Aggregating estimates by convex optimization

This page was built for publication: Deviation optimal learning using greedy \(Q\)-aggregation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q693750)