Deviation optimal learning using greedy Q-aggregation

DOI10.1214/12-AOS1025zbMATH Open1257.62037arXiv1203.2507MaRDI QIDQ693750FDOQ693750

Authors: Dong Dai, Philippe Rigollet, Tong Zhang 

Publication date: 10 December 2012

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: Given a finite family of functions, the goal of model selection aggregation is to construct a procedure that mimics the function from this family that is the closest to an unknown regression function. More precisely, we consider a general regression model with fixed design and measure the distance between functions by the mean squared error at the design points. While procedures based on exponential weights are known to solve the problem of model selection aggregation in expectation, they are, surprisingly, sub-optimal in deviation. We propose a new formulation called Q-aggregation that addresses this limitation; namely, its solution leads to sharp oracle inequalities that are optimal in a minimax sense. Moreover, based on the new formulation, we design greedy Q-aggregation procedures that produce sparse aggregation models achieving the optimal rate. The convergence and performance of these greedy procedures are illustrated and compared with other standard methods on simulated examples.

Full work available at URL: https://arxiv.org/abs/1203.2507

Recommendations

• Optimal learning with \textit{Q}-aggregation
• Hyper-sparse optimal aggregation
• Learning Theory and Kernel Machines
• Aggregation for Gaussian regression
• Optimal exponential bounds for aggregation of density estimators

zbMATH Keywords

model selectionregressionmodel averagingoracle inequalitiesgreedy algorithmlower boundsdeviation boundsexponential weightsdeviation suboptimality

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Nonparametric regression and quantile regression (62G08) Asymptotic properties of nonparametric inference (62G20) Methods of reduced gradient type (90C52)

Cites Work

• A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems
• Title not available (Why is that?)
• A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training
• Sharp oracle inequalities for aggregation of affine estimators
• Universal approximation bounds for superpositions of a sigmoidal function
• Kullback-Leibler aggregation and misspecified generalized linear models
• Title not available (Why is that?)
• Learning Theory and Kernel Machines
• Aggregation via empirical risk minimization
• Learning by mirror averaging
• Functional aggregation for nonparametric regression.
• Adaptive estimation of a quadratic functional by model selection.
• Title not available (Why is that?)
• Title not available (Why is that?)
• Exponential screening and optimal rates of sparse estimation
• Trading accuracy for sparsity in optimization problems with sparsity constraints
• Title not available (Why is that?)
• Hyper-sparse optimal aggregation
• On the optimality of the aggregate with exponential weights for low temperatures
• Aggregation by Exponential Weighting and Sharp Oracle Inequalities
• Aggregated estimators and empirical complexity for least square regression
• Title not available (Why is that?)

Cited In (22)

• Aggregation of affine estimators
• Sharp oracle inequalities for aggregation of affine estimators
• Hyper-sparse optimal aggregation
• Localized Gaussian width of \(M\)-convex hulls with applications to Lasso and convex aggregation
• Histopathological imaging‐based cancer heterogeneity analysis via penalized fusion with model averaging
• Optimal Kullback-Leibler aggregation in mixture density estimation by maximum likelihood
• Rank-Based Greedy Model Averaging for High-Dimensional Survival Data
• Optimal learning with \textit{Q}-aggregation
• Model aggregation for doubly divided data with large size and large dimension
• Solution of linear ill-posed problems by model selection and aggregation
• Optimal bounds for aggregation of affine estimators
• Statistical inference in compound functional models
• Martingale-residual-based greedy model averaging for high-dimensional current status data
• Title not available (Why is that?)
• Second-order Stein: SURE for SURE and other applications in high-dimensional inference
• Optimal learning with Bernstein online aggregation
• Optimal and Safe Estimation for High-Dimensional Semi-Supervised Learning
• An adaptive multiclass nearest neighbor classifier
• PAC-Bayesian risk bounds for group-analysis sparse regression by exponential weighting
• Estimation and Inference for High-Dimensional Generalized Linear Models with Knowledge Transfer
• Aggregating estimates by convex optimization
• User-friendly Introduction to PAC-Bayes Bounds

This page was built for publication: Deviation optimal learning using greedy \(Q\)-aggregation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q693750)