Adaptive Minimax Estimation over Sparse \ell_q-Hulls

MaRDI QIDQ2934056zbMATH OpenFDO

Authors Zhan Wang, Sandra Paterlini, Fuchang Gao, Yuhong Yang

Publication date 8 December 2014

Full work available at URL https://arxiv.org/abs/1108.1961, http://jmlr.csail.mit.edu/papers/v15/wang14b.html

zbMATH Keywords

high-dimensional sparse learningminimax rate of convergencemodel selectionoptimal aggregation, sparse \(q\)-constraint

Mathematics Subject Classification ID

Minimax procedures in statistical decision theory (62C20) Nonparametric regression and quantile regression (62G08) Ridge regression; shrinkage estimators (Lasso) (62J07)

Abstract: Given a dictionary of $M_n$ initial estimates of the unknown true regression function, we aim to construct linearly aggregated estimators that target the best performance among all the linear combinations under a sparse $q$-norm ($0leqqleq1$) constraint on the linear coefficients. Besides identifying the optimal rates of aggregation for these $el{l}_q$-aggregation problems, our multi-directional (or universal) aggregation strategies by model mixing or model selection achieve the optimal rates simultaneously over the full range of $0leqqleq1$ for general $M_n$ and upper bound $t_n$ of the $q$-norm. Both random and fixed designs, with known or unknown error variance, are handled, and the $el{l}_q$-aggregations examined in this work cover major types of aggregation problems previously studied in the literature. Consequences on minimax-rate adaptive regression under $el{l}_q$-constrained true coefficients ($0leqqleq1$) are also provided. Our results show that the minimax rate of $el{l}_q$-aggregation ($0leqqleq1$) is basically determined by an effective model size, which is a sparsity index that depends on $q$, $t_n$, $M_n$, and the sample size $n$ in an easily interpretable way based on a classical model selection theory that deals with a large number of models. In addition, in the fixed design case, the model selection approach is seen to yield optimal rates of convergence not only in expectation but also with exponential decay of deviation probability. In contrast, the model mixing approach can have leading constant one in front of the target risk in the oracle inequality while not offering optimality in deviation probability.

Recommendations

• High-Dimensional Adaptive Minimax Sparse Estimation With Interactions
• Nearly optimal minimax estimator for high-dimensional sparse linear regression
• Optimal adaptive estimation of linear functionals under sparsity
• Minimax adaptive dimension reduction for regression
• Minimax estimation of linear and quadratic functionals on sparsity classes
• Adaptive sparse regression
• Adaptive Lasso for sparse high-dimensional regression models
• Adaptive estimation in two-way sparse reduced-rank regression
• Sparse and robust estimation with ridge minimax concave penalty
• ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

Cited in

(29)

• A general procedure to combine estimators
• Minimax-rate adaptive nonparametric regression with unknown correlations of errors
• Optimal Kullback-Leibler aggregation in mixture density estimation by maximum likelihood
• Sharp oracle inequalities for aggregation of affine estimators
• On improvability of model selection by model averaging
• On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
• Model selection for high-dimensional linear regression with dependent observations
• A general framework for Bayes structured linear models
• Sharp oracle inequalities for low-complexity priors
• On Hodges' superefficiency and merits of oracle property in model selection
• Aggregation of affine estimators
• Minimax adaptive dimension reduction for regression
• Nearly optimal minimax estimator for high-dimensional sparse linear regression
• Regularization and the small-ball method. II: Complexity dependent error rates
• Aggregation of spectral density estimators
• Cross-validation for selecting a model selection procedure
• Performance Assessment of High-dimensional Variable Identification
• Jackknife model averaging for high‐dimensional quantile regression
• Optimal Integrating Learning for Split Questionnaire Design Type Data
• Low-rank matrix estimation via nonconvex optimization methods in multi-response errors-in-variables regression
• Information criteria for model selection
• Model averaging: a shrinkage perspective
• Sparse quantile regression via _0-penalty
• Characterizing the minimax rate of nonparametric regression under bounded star-shaped constraints
• On Optimality of Mallows Model Averaging
• Minimax rates of convergence for nonparametric location-scale models
• Sparse recovery via nonconvex regularized \(M\)-estimators over \(\ell_q\)-balls
• Toward optimal model averaging in regression models with time series errors
• Aggregation of predictors for nonstationary sub-linear processes and online adaptive forecasting of time varying autoregressive processes

This page was built for publication: Adaptive Minimax Estimation over Sparse $\ell_q$-Hulls

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2934056)