On Bayes risk lower bounds
From MaRDI portal
Publication:2953644
zbMATH Open1429.62046arXiv1410.0503MaRDI QIDQ2953644FDOQ2953644
Authors: Xi Chen, Adityanand Guntuboyina, Yuchen Zhang 
Publication date: 5 January 2017
Abstract: This paper provides a general technique for lower bounding the Bayes risk of statistical estimation, applicable to arbitrary loss functions and arbitrary prior distributions. A lower bound on the Bayes risk not only serves as a lower bound on the minimax risk, but also characterizes the fundamental limit of any estimator given the prior knowledge. Our bounds are based on the notion of -informativity, which is a function of the underlying class of probability measures and the prior. Application of our bounds requires upper bounds on the -informativity, thus we derive new upper bounds on -informativity which often lead to tight Bayes risk lower bounds. Our technique leads to generalizations of a variety of classical minimax bounds (e.g., generalized Fano's inequality). Our Bayes risk lower bounds can be directly applied to several concrete estimation problems, including Gaussian location models, generalized linear models, and principal component analysis for spiked covariance models. To further demonstrate the applications of our Bayes risk lower bounds to machine learning problems, we present two new theoretical results: (1) a precise characterization of the minimax risk of learning spherical Gaussian mixture models under the smoothed analysis framework, and (2) lower bounds for the Bayes risk under a natural prior for both the prediction and estimation errors for high-dimensional sparse linear regression under an improper learning setting.
Full work available at URL: https://arxiv.org/abs/1410.0503
Recommendations
- Bhattacharyya-type information inequality for the Bayes risk
- Information inequalities for the Bayes risk
- INFORMATION INEQUALITIES FOR THE MINIMAX RISK
- Lower bounds on the bayes risk eor statistical prediction problems
- A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation
Classification and discrimination; cluster analysis (statistical aspects) (62H30) Estimation in multivariate analysis (62H12) Bayesian problems; characterization of Bayes procedures (62C10) Minimax procedures in statistical decision theory (62C20) Measures of information, entropy (94A17)
Cited In (19)
- Fano's inequality for random variables
- Bayesian frequentist bounds for machine learning and system identification
- Bounds for Ratios of Posterior Expectations: Applications in the Collective Risk Model
- Lower bounds on the bayes risk eor statistical prediction problems
- On lower bounds for the bias-variance trade-off
- Learning Theory
- Statistical-computational trade-offs in tensor PCA and related problems via communication complexity
- A strong converse bound for multiple hypothesis testing, with applications to high-dimensional estimation
- A Bayesian view of the Hammersley–Chapman–Robbins-type inequality
- Information-theoretic upper and lower bounds for statistical estimation
- PAC-Bayesian bounds for randomized empirical risk minimizers
- Bayes Risk Error is a Bregman Divergence
- Biased Bayesian learning with an application to the risk-free rate puzzle
- Asymptotic Bayes risks for a general class of losses
- Risk bounds for statistical learning
- Obtaining minimax lower bounds: a review
- Lower bounds for the bayes risk In estimating a mixing parameter
- Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices
- Asymptotic Risk and Bayes Risk of Thresholding and Superefficient Estimates and Optimal Thresholding
This page was built for publication: On Bayes risk lower bounds
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2953644)
