Control of generalized error rates in multiple testing

DOI10.1214/009053606000001622MaRDI QIDQ155664zbMATH OpenOpenAlexFDO

Authors Joseph P. Romano, Michael Wolf, Joseph P Romano

Publication date 1 August 2007

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/0710.2258

zbMATH Keywords

bootstrapfalse discovery proportionfalse discovery rategeneralized familywise error ratestep-down procedure

Mathematics Subject Classification ID

Nonparametric statistical resampling methods (62G09) Nonparametric hypothesis testing (62G10) Asymptotic properties of nonparametric inference (62G20) Paired and multiple comparisons; multiple testing (62J15)

Abstract: Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $gamma$ and $alpha$, $PFDP>gammalealpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Recommendations

• Generalizations of the familywise error rate
• Balanced control of generalized error rates
• Stepup procedures for control of generalizations of the familywise error rate
• On stepdown control of the false discovery proportion
• Step-up and step-down procedures controlling the number and proportion of false positives

Cites work

• A Bootstrap Revival of Some Nonparametric Distance Tests
• A stochastic process approach to false discovery control.
• Balanced Simultaneous Confidence Sets
• Bootstrap methods in statistics
• Controlling the number of false discoveries: application to high-dimensional genomic data
• Empirical Bayes and Resampling Based Multiple Testing Procedure Controlling Tail Probability of the Proportion of False Positives.
• Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
• False Discovery Control for Random Fields
• Generalizations of the familywise error rate
• scientific article; zbMATH DE number 3624650 (Why is no real title available?)
• scientific article; zbMATH DE number 720689 (Why is no real title available?)
• scientific article; zbMATH DE number 1104922 (Why is no real title available?)
• scientific article; zbMATH DE number 838305 (Why is no real title available?)
• scientific article; zbMATH DE number 2220287 (Why is no real title available?)
• Multiple comparisons of biodiversity
• Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
• On optimality of stepdown and stepup multiple test procedures
• On stepdown control of the false discovery proportion
• Pairwise comparisons of the means of skewed data
• Prepivoting Test Statistics: A Bootstrap View of Asymptotic Refinements
• Resampling methods for dependent data
• Simulated power functions
• Some results on false discovery rate in stepwise multiple testing procedures.
• Stepup procedures for control of generalizations of the familywise error rate
• Stepwise Multiple Testing as Formalized Data Snooping
• Subsampling
• Testing Statistical Hypotheses
• The bootstrap and Edgeworth expansion
• The control of the false discovery rate in multiple testing under dependency.
• The jackknife and bootstrap

Cited in

(78)

• On a generalized false discovery rate
• Control of the false discovery rate under dependence using the bootstrap and subsampling
• Rejoinder on: Control of the false discovery rate under dependence using the bootstrap and subsampling
• Step-up and step-down procedures controlling the number and proportion of false positives
• Exceedance Control of the False Discovery Proportion
• StepwiseTest
• A modified Benjamini-Hochberg multiple comparisons procedure for controlling the false discovery rate
• Statistical procedures for the market graph construction
• Sequential tests controlling generalized familywise error rates
• Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics.
• The control of the false discovery rate in multiple testing under dependency.
• Multiple hypotheses testing and expected number of type I errors
• Asymptotic properties of false discovery rate controlling procedures under independence
• On stepwise control of the generalized familywise error rate
• Only closed testing procedures are admissible for controlling false discovery proportions
• Distributions associated with simultaneous multiple hypothesis testing
• Semi-supervised multiple testing
• A comprehensive error rate for multiple testing
• Controlling the false discovery exceedance for heterogeneous tests
• Outcome-wide longitudinal designs for causal inference: a new template for empirical studies
• A sufficient criterion for control of some generalized error rates in multiple testing
• Nonparametric false discovery rate control for identifying simultaneous signals
• Classes of multiple decision functions strongly controlling FWER and FDR
• New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
• Coarse-to-fine multiple testing strategies
• Simultaneous control of false positives and false negatives in multiple hypotheses testing
• Generalizations of the familywise error rate
• New multiple testing method under no dependency assumption, with application to multiple comparisons problem
• Estimating minimum effect with outlier selection
• Familywise error rate control via knockoffs
• Bonferroni-type Plug-in Procedure Controlling Generalized Familywise Error Rate
• Multiple Comparisons Controlling Expected Number of False Discoveries
• k-FWER control without \(p \)-value adjustment, with application to detection of genetic determinants of multiple sclerosis in Italian twins
• Statistical properties of an early stopping rule for resampling-based multiple testing
• False Discovery Control for Multiple Tests of Association Under General Dependence
• Rotation-based multiple testing in the multivariate linear model
• Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing
• Procedures controlling the k-FDR using bivariate distributions of the null p-values
• FORMALIZED DATA SNOOPING BASED ON GENERALIZED ERROR RATES
• A note on control of the false discovery proportion
• Step-up and step-down methods for testing multiple hypotheses in sequential experiments
• A method for modifying multiple testing procedures
• Step-up procedure controlling generalized family-wise error rate
• Stepup procedures for control of generalizations of the familywise error rate
• Comment
• Controlling individual and experimentwise error rates in replicated regular two-level factorial experiments
• Evaluations of FWER-controlling methods in multiple hypothesis testing
• Fast closed testing for exchangeable local tests
• Control of Directional Errors in Fixed Sequence Multiple Testing
• Valid post-selection inference in high-dimensional approximately sparse quantile regression models
• Bootstrap Joint Prediction Regions
• Type I error rate control for testing many hypotheses: a survey with proofs
• On stepdown control of the false discovery proportion
• \(F\)-distribution calibrated empirical likelihood ratio tests for multiple hypothesis testing
• A Generalized Sidak-Holm Procedure and Control of Generalized Error Rates under Independence
• A Note on Controlling the Number of False Positives
• Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
• Type I error control and the power to detect factorial effects
• Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• Multiple testing for exploratory research
• Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari
• Threshold Selection in Feature Screening for Error Rate Control
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• Confidence and discoveries with e-values
• Stepdown Procedures Controlling a Generalized False Discovery Rate
• A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
• Simultaneous critical values for t-tests in very high dimensions
• A comment on: ``Invidious comparisons: ranking and selection as compound decisions by Jiaying Gu and Roger Koenker
• Competition-based control of the false discovery proportion
• Statistical nonparametric mapping: multivariate permutation tests for location, correlation, and regression problems in neuroimaging
• An Empirical Bayes Approach to Controlling the False Discovery Exceedance
• Graphical approaches for the control of generalized error rates
• Flexible control of the median of the false discovery proportion
• Regularizing stock return covariance matrices via multiple testing of correlations
• Neural networks in financial trading
• Balanced control of generalized error rates
• On methods controlling the false discovery rate

Describes a project that uses

Uses Software

• boot

This page was built for publication: Control of generalized error rates in multiple testing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q155664)