Efficient, adaptive cross-validation for tuning and comparing models, with application to drug discovery

DOI10.1214/11-AOAS491zbMATH Open1234.62115arXiv1202.6536OpenAlexW3101073625WikidataQ57434099 ScholiaQ57434099MaRDI QIDQ766007FDOQ766007

Authors: Hui Shen, William J. Welch, Jacqueline M. Hughes-Oliver 

Publication date: 22 March 2012

Published in: The Annals of Applied Statistics (Search for Journal in Brave)

Abstract: Cross-validation (CV) is widely used for tuning a model with respect to user-selected parameters and for selecting a "best" model. For example, the method of $k$-nearest neighbors requires the user to choose $k$, the number of neighbors, and a neural network has several tuning parameters controlling the network complexity. Once such parameters are optimized for a particular data set, the next step is often to compare the various optimized models and choose the method with the best predictive performance. Both tuning and model selection boil down to comparing models, either across different values of the tuning parameters or across different classes of statistical models and/or sets of explanatory variables. For multiple large sets of data, like the PubChem drug discovery cheminformatics data which motivated this work, reliable CV comparisons are computationally demanding, or even infeasible. In this paper we develop an efficient sequential methodology for model comparison based on CV. It also takes into account the randomness in CV. The number of models is reduced via an adaptive, multiplicity-adjusted sequential algorithm, where poor performers are quickly eliminated. By exploiting matching of individual observations, it is sometimes even possible to establish the statistically significant inferiority of some models with just one execution of CV.

Full work available at URL: https://arxiv.org/abs/1202.6536

Recommendations

• Cross-validation with confidence
• Fast cross-validation via sequential testing
• Cross-validation for selecting a model selection procedure
• Multiple predicting \(K\)-fold cross-validation for model selection
• Consistency of cross validation for comparing regression procedures

zbMATH Keywords

sequential analysisneural networks\(k\)-nearest neighborsmultiplicity adjustmentassay datacheminformaticsPubChemrandomized-block design

Mathematics Subject Classification ID

Applications of statistics to biology and medical sciences; meta analysis (62P10) Sequential statistical analysis (62L10)

Cites Work

• A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods
• Title not available (Why is that?)
• Model selection via multifold cross validation
• Linear Model Selection by Cross-Validation
• Asymptotics of cross-validated risk estimation in estimator selection and performance assess\-ment
• Title not available (Why is that?)
• Title not available (Why is that?)
• Asymptotic optimality for \(C_ p\), \(C_ L\), cross-validation and generalized cross-validation: Discrete index set
• Asymptotics for and against cross-validation
• Title not available (Why is that?)
• Title not available (Why is that?)
• Deletion/Substitution/Addition Algorithm in Learning with Applications in Genomics
• Design and Analysis of Experiments

Cited In (2)

• Fast cross-validation via sequential testing
• Cross-validation with confidence

This page was built for publication: Efficient, adaptive cross-validation for tuning and comparing models, with application to drug discovery

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q766007)