Conditional predictive inference for stable algorithms

DOI10.1214/22-AOS2250arXiv1809.01412OpenAlexW4360879309MaRDI QIDQ6042347FDOQ6042347

Authors: Lukas Steinberger, Hannes Leeb 

Publication date: 10 May 2023

Published in: The Annals of Statistics (Search for Journal in Brave)

Abstract: We investigate generically applicable and intuitively appealing prediction intervals based on $k$-fold cross validation. We focus on the conditional coverage probability of the proposed intervals, given the observations in the training sample (hence, training conditional validity), and show that it is close to the nominal level, in an appropriate sense, provided that the underlying algorithm used for computing point predictions is sufficiently stable when feature-response pairs are omitted. Our results are based on a finite sample analysis of the empirical distribution function of $k$-fold cross validation residuals and hold in non-parametric settings with only minimal assumptions on the error distribution. To illustrate our results, we also apply them to high-dimensional linear predictors, where we obtain uniform asymptotic training conditional validity as both sample size and dimension tend to infinity at the same rate and consistent parameter estimation typically fails. These results show that despite the serious problems of resampling procedures for inference on the unknown parameters (cf. Bickel and Freedman, 1983; El Karoui and Purdom, 2018; Mammen, 1996), cross validation methods can be successfully applied to obtain reliable predictive inference even in high dimensions and conditionally on the training data.

Full work available at URL: https://arxiv.org/abs/1809.01412

Recommendations

• Predictive inference with the jackknife+
• Conditional predictive inference post model selection
• Bootstrap prediction intervals with asymptotic conditional validity and unconditional guarantees
• The limits of distribution-free conditional predictive inference
• Cross-validation with confidence

zbMATH Keywords

cross-validationhigh-dimensional regressionprediction intervalsalgorithmic stability

Mathematics Subject Classification ID

Asymptotic properties of nonparametric inference (62G20) General nonlinear regression (62J02) Nonparametric tolerance and confidence regions (62G15) Ridge regression; shrinkage estimators (Lasso) (62J07)

Cites Work

• On-line predictive linear regression
• Consistency of random forests
• Statistics for high-dimensional data. Methods, theory and applications.
• Title not available (Why is that?)
• Title not available (Why is that?)
• A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity
• Predictive inference with the jackknife+
• Distribution-free predictive inference for regression
• Bootstrap Prediction Intervals for Regression
• Predictive Intervals Based on Reuse of the Sample
• Distribution-free inequalities for the deleted and holdout error estimates
• Title not available (Why is that?)
• 10.1162/153244302760200704
• Fast exact conformalization of the Lasso using piecewise linear homotopy
• Conditional validity of inductive conformal predictors
• Distribution-Free Prediction Sets
• Distribution-free Prediction Bands for Non-parametric Regression
• A distribution-free theory of nonparametric regression
• Using Least Squares to Approximate Unknown Regression Functions
• Model-free model-fitting and predictive distributions
• Statistical Tolerance Regions: Theory, Applications, and Computation
• Prediction intervals for regression models
• Maximum likelihood estimation in misspecified generalized linear models
• Title not available (Why is that?)
• Asymptotically Valid Prediction Intervals for Linear Models
• The spectrum of kernel random matrices
• Shrinkage estimators for prediction out-of-sample: conditional performance
• On robust regression with high-dimensional predictors
• Optimal equivariant prediction for high-dimensional linear models with arbitrary predictor covariance
• Non-Parametric Estimation II. Statistically Equivalent Blocks and Tolerance Regions--The Continuous Case
• Empirical process of residuals for high-dimensional linear models
• Asymptotically Minimal Multivariate Tolerance Sets
• Multivariate spacings based on data depth. I: Construction of nonparametric multivariate tolerance regions
• Smallest nonparametric tolerance regions.
• Can we trust the bootstrap in high-dimensions? The case of linear models
• On the impact of predictor geometry on the performance on high-dimensional ridge-regularized generalized robust regression estimators
• Statistical Prediction with Special Reference to the Problem of Tolerance Limits
• An Extension of Wilks' Method for Setting Tolerance Limits
• Nonparametric regression using deep neural networks with ReLU activation function
• The limits of distribution-free conditional predictive inference

Cited In (6)

• Title not available (Why is that?)
• Conformal prediction: a unified review of theory and new challenges
• Post-selection inference via algorithmic stability
• The limits of distribution-free conditional predictive inference
• Training-conditional coverage for distribution-free predictive inference
• Conditional validity of inductive conformal predictors

This page was built for publication: Conditional predictive inference for stable algorithms

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6042347)