A survey of cross-validation procedures for model selection

DOI10.1214/09-SS054zbMATH Open1190.62080arXiv0907.4728OpenAlexW3104887532WikidataQ56689497 ScholiaQ56689497MaRDI QIDQ975579FDOQ975579

Authors: Sylvain Arlot, Alain Celisse 

Publication date: 9 June 2010

Published in: Statistics Surveys (Search for Journal in Brave)

Abstract: Used to estimate the risk of an estimator or to perform model selection, cross-validation is a widespread strategy because of its simplicity and its apparent universality. Many results exist on the model selection performances of cross-validation procedures. This survey intends to relate these results to the most recent advances of model selection theory, with a particular emphasis on distinguishing empirical statements from rigorous theoretical results. As a conclusion, guidelines are provided for choosing the best cross-validation procedure according to the particular features of the problem in hand.

Full work available at URL: https://arxiv.org/abs/0907.4728

Recommendations

• Cross-validation for selecting a model selection procedure
• Asymptotics of cross-validated risk estimation in estimator selection and performance assess\-ment
• Linear Model Selection by Cross-Validation
• Model selection via multifold cross validation
• Oracle inequalities for multi-fold cross validation

zbMATH Keywords

model selectioncross-validationleave-one-out

Mathematics Subject Classification ID

Nonparametric estimation (62G05) Nonparametric regression and quantile regression (62G08) Nonparametric inference (62G99)

Cites Work

• On the bias and variability of bootstrap and cross-validation estimates of error rate in discrimination problems
• Optimal Oracle Inequality for Aggregation of Classifiers Under Low Noise Condition
• Title not available (Why is that?)
• Estimating the dimension of a model
• Heuristics of instability and stabilization in model selection
• Bayesian model averaging: A tutorial. (with comments and a rejoinder).
• Title not available (Why is that?)
• A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• Some Comments on C P
• An asymptotically optimal window selection rule for kernel density estimates
• Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation
• Improvements on Cross-Validation: The .632+ Bootstrap Method
• Title not available (Why is that?)
• Robust Estimation of a Location Parameter
• Model selection via multifold cross validation
• Consistency of cross validation for comparing regression procedures
• Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation
• Title not available (Why is that?)
• Title not available (Why is that?)
• Linear Model Selection by Cross-Validation
• Histogram selection in non Gaussian regression
• Gaussian model selection
• Segmentation of the mean of heteroscedastic data via cross-validation
• Asymptotic properties of criteria for selection of variables in multiple regression
• Least angle and \(\ell _{1}\) penalized regression: a review
• Title not available (Why is that?)
• The Predictive Sample Reuse Method with Applications
• 10.1162/153244302760200704
• Title not available (Why is that?)
• Title not available (Why is that?)
• Risk bounds for model selection via penalization
• Approximate efficiency of a selection procedure for the number of regression variables
• How Biased is the Apparent Error Rate of a Prediction Rule?
• Kernel Regression Estimation Using Repeated Measurements Data
• Title not available (Why is that?)
• Title not available (Why is that?)
• Model Selection and Multimodel Inference
• Title not available (Why is that?)
• Asymptotic optimality for \(C_ p\), \(C_ L\), cross-validation and generalized cross-validation: Discrete index set
• On Kullback-Leibler loss and density estimation
• Smoothed cross-validation
• Model selection in nonparametric regression
• Inference for the generalization error
• Nonparametric regression with correlated errors.
• A distribution-free theory of nonparametric regression
• Model selection by resampling penalization
• No unbiased estimator of the variance of K-fold cross-validation
• DATA-DEPENDENT ESTIMATION OF PREDICTION FUNCTIONS
• A cross-validatory method for dependent data
• Adaptive Regression by Mixing
• The Relationship between Variable Selection and Data Agumentation and a Method for Prediction
• Comparison of two bandwidth selectors with dependent errors
• Minimal penalties for Gaussian model selection
• Title not available (Why is that?)
• Concentration inequalities and model selection. Ecole d'Eté de Probabilités de Saint-Flour XXXIII -- 2003.
• A universal prior for integers and estimation by minimum description length
• The Estimation of Prediction Error
• Consistent cross-validated density estimation
• Estimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation
• Robust Linear Model Selection by Cross-Validation
• Data-driven bandwidth choice for density estimation based on dependent data
• How Far Are Automatically Chosen Regression Smoothing Parameters From Their Optimum?
• Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy
• Large sample optimality of least squares cross-validation in density estimation
• Estimation of dependences based on empirical data. Transl. from the Russian by Samuel Kotz
• Statistical predictor identification
• Rademacher penalties and structural risk minimization
• Title not available (Why is that?)
• Model selection for regression on a random design
• Theory of Classification: a Survey of Some Recent Advances
• Asymptotics for and against cross-validation
• Weak convergence of dependent empirical measures with application to subsampling in function spaces
• Cross-validation in nonparametric regression with outliers
• Bandwidth selection in robust smoothing
• Oracle inequalities for multi-fold cross validation
• The cross-validated adaptive epsilon-net estimator
• Cross-Validation of Regression Models
• Bootstrap Model Selection
• Title not available (Why is that?)
• Model selection and error estimation
• A local cross-validation algorithm
• Distribution-free performance bounds for potential function rules
• From Stein's unbiased risk estimates to the method of generalized cross- validation
• Periodic splines for spectral density estimation: the use of cross validation for determining the degree of smoothing
• Suboptimality of Penalized Empirical Risk Minimization in Classification
• Title not available (Why is that?)
• Title not available (Why is that?)
• Title not available (Why is that?)
• On bandwidth choice in nonparametric regression with both short- and long-range dependent errors
• Title not available (Why is that?)
• Title not available (Why is that?)
• Nonparametric density estimation by exact leave-\(p\)-out cross-validation
• A predictive approach to the random effect model
• Title not available (Why is that?)
• Cross validation model selection criteria for linear regression based on the Kullback-Leibler discrepancy
• Title not available (Why is that?)
• Asymptotic comparison of (partial) cross-validation, GCV and randomized GCV in nonparametric regression
• Estimation of the conditional risk in classification: the swapping method
• Analysis of variance of cross-validation estimators of the generalization error
• Title not available (Why is that?)

Cited In (only showing first 100 items - show all)

• Bayesian functional linear regression with sparse step functions
• On cross-validated Lasso in high dimensions
• A cross-validation based estimation of the proportion of true null hypotheses
• A survey of Bayesian predictive methods for model assessment, selection and comparison
• Cross-Validation--based Adaptive Sampling for Gaussian Process Models
• Oracle inequalities for multi-fold cross validation
• Parallel cross-validation: a scalable fitting method for Gaussian process models
• On the usefulness of cross-validation for directional forecast evaluation
• Data science, big data and statistics
• Bayesian calibration, validation and uncertainty quantification for predictive modelling of tumour growth: a tutorial
• Portfolio approaches for constraint optimization problems
• On polynomial chaos expansion via gradient-enhanced \(\ell_1\)-minimization
• Generalised density forecast combinations
• Model selection in reinforcement learning
• Virtual model validation of complex multiscale systems: applications to nonlinear elastostatics
• The adaptive and the thresholded Lasso for potentially misspecified models (and a lower bound for the Lasso)
• Consistency of cross validation for comparing regression procedures
• Semiparametric zero-inflated modeling in multi-ethnic study of atherosclerosis (MESA)
• Oracle inequalities for cross-validation type procedures
• Estimating the Kullback–Liebler risk based on multifold cross‐validation
• Robust machine learning by median-of-means: theory and practice
• Modeling uncertainty of expert elicitation for use in risk-based optimization
• Ranking in evolving complex networks
• Generative adversarial networks for financial trading strategies fine-tuning and combination
• Spectral likelihood expansions for Bayesian inference
• Prediction error bounds for linear regression with the TREX
• Leave-one-out cross-validation is risk consistent for Lasso
• A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
• Decision-based model selection
• Data partition methodology for validation of predictive models
• Model selection properties of forward selection and sequential cross‐validation for high‐dimensional regression
• Estimation of nonbinary random response
• Global resolution of the support vector machine regression parameters selection problem with LPCC
• Penalized likelihood methods for modeling count data
• Support vector regression based on grid-search method for short-term wind power forecasting
• A new technique for postsample model selection and validation
• Use of a novel grammatical inference approach in classification of amyloidogenic hexapeptides
• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• A note on the validity of cross-validation for evaluating autoregressive time series prediction
• Estimator selection in the Gaussian setting
• R package rjmcmc: reversible jump MCMC using post‐processing
• Efficient approximate \(k\)-fold and leave-one-out cross-validation for ridge regression
• Hierarchical Bayesian models of reinforcement learning: introduction and comparison to alternative methods
• Extensions of stability selection using subsamples of observations and covariates
• Physically-constrained data-driven inversions to infer the bed topography beneath glaciers flows. Application to East Antarctica
• Optimal cross-validation in density estimation with the \(L^{2}\)-loss
• Slope heuristics: overview and implementation
• Rank adaptive tensor recovery based model reduction for partial differential equations with high-dimensional random inputs
• SSC-EKE: semi-supervised classification with extensive knowledge exploitation
• Compressive sampling of polynomial chaos expansions: convergence analysis and sampling strategies
• Linear Model Selection by Cross-Validation
• Framing reinforcement learning from human reward: reward positivity, temporal discounting, episodicity, and performance
• Fast Cross-validation for Multi-penalty High-dimensional Ridge Regression
• Asymptotics for regression models under loss of identifiability
• Semiparametric stochastic volatility modelling using penalized splines
• Cross-Validation, Risk Estimation, and Model Selection: Comment on a Paper by Rosset and Tibshirani
• Segmentation of the mean of heteroscedastic data via cross-validation
• A comparison of regularization methods applied to the linear discriminant function with high-dimensional microarray data
• Minimization and estimation of the variance of prediction errors for cross-validation designs
• Derivation, identification and validation of a computational model of a novel synthetic regulatory network in yeast
• Estimating the trace of the matrix inverse by interpolating from the diagonal of an approximate inverse
• A survey on learning approaches for undirected graphical models. Application to scene object recognition
• Data-driven robust optimization
• SUNNY: a lazy portfolio approach for constraint solving
• An Empirical Study of Indirect Cross-Validation
• Cross-validation for selecting a model selection procedure
• High-dimensional regression with unknown variance
• Data-based models for the prediction of dam behaviour: a review and some methodological considerations
• Prediction of arch dam deformation via correlated multi-target stacking
• A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
• From zero to hero: realized partial (co)variances
• An algorithm for computationally expensive engineering optimization problems
• Targeted cross-validation
• Title not available (Why is that?)
• Clusterwise linear regression modeling with soft scale constraints
• Metamodel construction for sensitivity analysis
• Modelling of count data using nonparametric mixtures
• Efficient semiparametric estimation and model selection for multidimensional mixtures
• Cross-validation based weights and structure determination of Chebyshev-polynomial neural networks for pattern classification
• Projection-based text line segmentation with a variable threshold
• Equity clusters through the lens of realized semicorrelations
• State-by-state Minimax Adaptive Estimation for Nonparametric Hidden Markov Models
• Low-rank approximation and completion of positive tensors
• An augmented MFS approach for brain activity reconstruction
• MLP-ANN-based execution time prediction model and assessment of input parameters through structural modeling
• Clustered active-subspace based local Gaussian process emulator for high-dimensional and complex computer models
• Aggregated hold out for sparse linear regression with a robust loss function
• Modeling strength and failure variability due to porosity in additively manufactured metals
• Model selection in utility-maximizing binary prediction
• Variational approach for learning Markov processes from time series data
• A data driven equivariant approach to constrained Gaussian mixture modeling
• A new approach for classifier model selection and tuning using logistic regression and genetic algorithms
• Fermentation of \textit{Saccharomyces cerevisiae} -- combining kinetic modeling and optimization techniques points out avenues to effective process design
• Model selection criteria based on cross-validatory concordance statistics
• \textsf{StreaMRAK} a streaming multi-resolution adaptive kernel algorithm
• A MOM-based ensemble method for robustness, subsampling and hyperparameter tuning
• Comorbidity of chronic diseases in the elderly: patterns identified by a copula design for mixed responses
• Bayesian inversion using adaptive polynomial chaos kriging within subset simulation
• Comprehensive analysis of gradient-based hyperparameter optimization algorithms
• Estimation of the global mode of a density: minimaxity, adaptation, and computational complexity

Uses Software

• ElemStatLearn

This page was built for publication: A survey of cross-validation procedures for model selection

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q975579)