Control of generalized error rates in multiple testing

DOI10.1214/009053606000001622MaRDI QIDQ155664zbMATHOpenAlexFDO

Authors Joseph P. Romano, Michael Wolf, Michael Wolf, Joseph P Romano

Publication date 1 August 2007

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/0710.2258

zbMATH Keywords

bootstrapfalse discovery ratefalse discovery proportiongeneralized familywise error ratestep-down procedure

Mathematics Subject Classification ID

Nonparametric hypothesis testing (62G10) Asymptotic properties of nonparametric inference (62G20) Nonparametric statistical resampling methods (62G09) Paired and multiple comparisons; multiple testing (62J15)

Abstract: Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $gamma$ and $alpha$, $PFDP>gammalealpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Recommendations

• Generalizations of the familywise error rate
• Balanced control of generalized error rates
• Stepup procedures for control of generalizations of the familywise error rate
• On stepdown control of the false discovery proportion
• Step-up and step-down procedures controlling the number and proportion of false positives

Cites work

• scientific article; zbMATH DE number 3624650 (Why is no real title available?)
• scientific article; zbMATH DE number 720689 (Why is no real title available?)
• scientific article; zbMATH DE number 1104922 (Why is no real title available?)
• scientific article; zbMATH DE number 838305 (Why is no real title available?)
• scientific article; zbMATH DE number 2220287 (Why is no real title available?)
• A Bootstrap Revival of Some Nonparametric Distance Tests
• A stochastic process approach to false discovery control.
• Balanced Simultaneous Confidence Sets
• Bootstrap methods in statistics
• Controlling the number of false discoveries: application to high-dimensional genomic data
• Empirical Bayes and Resampling Based Multiple Testing Procedure Controlling Tail Probability of the Proportion of False Positives.
• Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
• False Discovery Control for Random Fields
• Generalizations of the familywise error rate
• Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
• Multiple comparisons of biodiversity
• On optimality of stepdown and stepup multiple test procedures
• On stepdown control of the false discovery proportion
• Pairwise comparisons of the means of skewed data
• Prepivoting Test Statistics: A Bootstrap View of Asymptotic Refinements
• Resampling methods for dependent data
• Simulated power functions
• Some results on false discovery rate in stepwise multiple testing procedures.
• Stepup procedures for control of generalizations of the familywise error rate
• Stepwise Multiple Testing as Formalized Data Snooping
• Subsampling
• Testing Statistical Hypotheses
• The bootstrap and Edgeworth expansion
• The control of the false discovery rate in multiple testing under dependency.
• The jackknife and bootstrap

Cited in

(76)

• On stepwise control of the generalized familywise error rate
• Classes of multiple decision functions strongly controlling FWER and FDR
• Sequential tests controlling generalized familywise error rates
• Only closed testing procedures are admissible for controlling false discovery proportions
• Familywise error rate control via knockoffs
• Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
• Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics.
• A modified Benjamini-Hochberg multiple comparisons procedure for controlling the false discovery rate
• Generalizations of the familywise error rate
• Neural networks in financial trading
• Controlling the false discovery exceedance for heterogeneous tests
• Statistical properties of an early stopping rule for resampling-based multiple testing
• Outcome-wide longitudinal designs for causal inference: a new template for empirical studies
• Rotation-based multiple testing in the multivariate linear model
• Step-up procedure controlling generalized family-wise error rate
• Simultaneous control of false positives and false negatives in multiple hypotheses testing
• A comprehensive error rate for multiple testing
• False Discovery Control for Multiple Tests of Association Under General Dependence
• Multiple hypotheses testing and expected number of type I errors
• Procedures controlling the \(k\)-FDR using bivariate distributions of the null \(p\)-values
• Semi-supervised multiple testing
• Threshold Selection in Feature Screening for Error Rate Control
• The control of the false discovery rate in multiple testing under dependency.
• Multiple testing for exploratory research
• Bonferroni-type Plug-in Procedure Controlling Generalized Familywise Error Rate
• A method for modifying multiple testing procedures
• Step-up and step-down methods for testing multiple hypotheses in sequential experiments
• Simultaneous critical values for \(t\)-tests in very high dimensions
• Control of Directional Errors in Fixed Sequence Multiple Testing
• Type I error rate control for testing many hypotheses: a survey with proofs
• Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• On a generalized false discovery rate
• k-FWER control without \(p \)-value adjustment, with application to detection of genetic determinants of multiple sclerosis in Italian twins
• A sufficient criterion for control of some generalized error rates in multiple testing
• FORMALIZED DATA SNOOPING BASED ON GENERALIZED ERROR RATES
• A Generalized Sidak-Holm Procedure and Control of Generalized Error Rates under Independence
• Evaluations of FWER-controlling methods in multiple hypothesis testing
• Coarse-to-fine multiple testing strategies
• Bootstrap Joint Prediction Regions
• Step-up and step-down procedures controlling the number and proportion of false positives
• Multiple Comparisons Controlling Expected Number of False Discoveries
• StepwiseTest
• \(F\)-distribution calibrated empirical likelihood ratio tests for multiple hypothesis testing
• Control of the false discovery rate under dependence using the bootstrap and subsampling
• Statistical procedures for the market graph construction
• A Note on Controlling the Number of False Positives
• Fast closed testing for exchangeable local tests
• New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
• Estimating minimum effect with outlier selection
• Valid post-selection inference in high-dimensional approximately sparse quantile regression models
• Balanced control of generalized error rates
• On stepdown control of the false discovery proportion
• Stepup procedures for control of generalizations of the familywise error rate
• A note on control of the false discovery proportion
• On methods controlling the false discovery rate
• New multiple testing method under no dependency assumption, with application to multiple comparisons problem
• Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
• Confidence and discoveries with \(e\)-values
• Nonparametric false discovery rate control for identifying simultaneous signals
• Distributions associated with simultaneous multiple hypothesis testing
• Exceedance Control of the False Discovery Proportion
• Asymptotic properties of false discovery rate controlling procedures under independence
• Rejoinder on: Control of the false discovery rate under dependence using the bootstrap and subsampling
• An Empirical Bayes Approach to Controlling the False Discovery Exceedance
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Type I error control and the power to detect factorial effects
• Statistical nonparametric mapping: multivariate permutation tests for location, correlation, and regression problems in neuroimaging
• A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
• Controlling individual and experimentwise error rates in replicated regular two-level factorial experiments
• A comment on: ``Invidious comparisons: ranking and selection as compound decisions by Jiaying Gu and Roger Koenker
• Competition-based control of the false discovery proportion
• Stepdown Procedures Controlling a Generalized False Discovery Rate
• Graphical approaches for the control of generalized error rates
• Comment
• Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari

Describes a project that uses

Uses Software

• boot

This page was built for publication: Control of generalized error rates in multiple testing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q155664)