Control of generalized error rates in multiple testing

DOI10.1214/009053606000001622MaRDI QIDQ155664zbMATH OpenOpenAlexFDO

Authors Joseph P. Romano, Michael Wolf, Joseph P Romano

Publication date 1 August 2007

Published in The Annals of Statistics (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/0710.2258

zbMATH Keywords

bootstrapfalse discovery ratefalse discovery proportiongeneralized familywise error ratestep-down procedure

Mathematics Subject Classification ID

Nonparametric hypothesis testing (62G10) Asymptotic properties of nonparametric inference (62G20) Nonparametric statistical resampling methods (62G09) Paired and multiple comparisons; multiple testing (62J15)

Abstract: Consider the problem of testing $s$ hypotheses simultaneously. The usual approach restricts attention to procedures that control the probability of even one false rejection, the familywise error rate (FWER). If $s$ is large, one might be willing to tolerate more than one false rejection, thereby increasing the ability of the procedure to correctly reject false null hypotheses. One possibility is to replace control of the FWER by control of the probability of $k$ or more false rejections, which is called the $k$-FWER. We derive both single-step and step-down procedures that control the $k$-FWER in finite samples or asymptotically, depending on the situation. We also consider the false discovery proportion (FDP) defined as the number of false rejections divided by the total number of rejections (and defined to be 0 if there are no rejections). The false discovery rate proposed by Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B 57 (1995) 289--300] controls $E(FDP)$. Here, the goal is to construct methods which satisfy, for a given $gamma$ and $alpha$, $PFDP>gammalealpha$, at least asymptotically. In contrast to the proposals of Lehmann and Romano [Ann. Statist. 33 (2005) 1138--1154], we construct methods that implicitly take into account the dependence structure of the individual test statistics in order to further increase the ability to detect false null hypotheses. This feature is also shared by related work of van der Laan, Dudoit and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 15], but our methodology is quite different. Like the work of Pollard and van der Laan [Proc. 2003 International Multi-Conference in Computer Science and Engineering, METMBS'03 Conference (2003) 3--9] and Dudoit, van der Laan and Pollard [Stat. Appl. Genet. Mol. Biol. 3 (2004) article 13], we employ resampling methods to achieve our goals. Some simulations compare finite sample performance to currently available methods.

Recommendations

• Generalizations of the familywise error rate
• Balanced control of generalized error rates
• Stepup procedures for control of generalizations of the familywise error rate
• On stepdown control of the false discovery proportion
• Step-up and step-down procedures controlling the number and proportion of false positives

Cites work

• scientific article; zbMATH DE number 3624650 (Why is no real title available?)
• scientific article; zbMATH DE number 720689 (Why is no real title available?)
• scientific article; zbMATH DE number 1104922 (Why is no real title available?)
• scientific article; zbMATH DE number 838305 (Why is no real title available?)
• scientific article; zbMATH DE number 2220287 (Why is no real title available?)
• A Bootstrap Revival of Some Nonparametric Distance Tests
• A stochastic process approach to false discovery control.
• Balanced Simultaneous Confidence Sets
• Bootstrap methods in statistics
• Controlling the number of false discoveries: application to high-dimensional genomic data
• Empirical Bayes and Resampling Based Multiple Testing Procedure Controlling Tail Probability of the Proportion of False Positives.
• Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
• False Discovery Control for Random Fields
• Generalizations of the familywise error rate
• Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
• Multiple comparisons of biodiversity
• On optimality of stepdown and stepup multiple test procedures
• On stepdown control of the false discovery proportion
• Pairwise comparisons of the means of skewed data
• Prepivoting Test Statistics: A Bootstrap View of Asymptotic Refinements
• Resampling methods for dependent data
• Simulated power functions
• Some results on false discovery rate in stepwise multiple testing procedures.
• Stepup procedures for control of generalizations of the familywise error rate
• Stepwise Multiple Testing as Formalized Data Snooping
• Subsampling
• Testing Statistical Hypotheses
• The bootstrap and Edgeworth expansion
• The control of the false discovery rate in multiple testing under dependency.
• The jackknife and bootstrap

Cited in

(78)

• Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari
• Graphical approaches for the control of generalized error rates
• A comprehensive error rate for multiple testing
• Bonferroni-type Plug-in Procedure Controlling Generalized Familywise Error Rate
• Control of the false discovery rate under dependence using the bootstrap and subsampling
• Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates
• Multiple testing for exploratory research
• Statistical properties of an early stopping rule for resampling-based multiple testing
• New multiple testing method under no dependency assumption, with application to multiple comparisons problem
• Competition-based control of the false discovery proportion
• Estimating minimum effect with outlier selection
• Neural networks in financial trading
• Nonparametric false discovery rate control for identifying simultaneous signals
• Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics.
• The control of the false discovery rate in multiple testing under dependency.
• On methods controlling the false discovery rate
• Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing
• Type I error control and the power to detect factorial effects
• Sequential tests controlling generalized familywise error rates
• \(F\)-distribution calibrated empirical likelihood ratio tests for multiple hypothesis testing
• Familywise error rate control via knockoffs
• A method for modifying multiple testing procedures
• Step-up and step-down methods for testing multiple hypotheses in sequential experiments
• Simultaneous control of false positives and false negatives in multiple hypotheses testing
• Anomaly Detection for a Large Number of Streams: A Permutation-Based Higher Criticism Approach
• A Generalized Sidak-Holm Procedure and Control of Generalized Error Rates under Independence
• Valid post-selection inference in high-dimensional approximately sparse quantile regression models
• Asymptotic properties of false discovery rate controlling procedures under independence
• Multiple hypotheses testing and expected number of type I errors
• Generalizations of the familywise error rate
• On stepdown control of the false discovery proportion
• Step-up procedure controlling generalized family-wise error rate
• On stepwise control of the generalized familywise error rate
• On a generalized false discovery rate
• Rotation-based multiple testing in the multivariate linear model
• Flexible control of the median of the false discovery proportion
• Regularizing stock return covariance matrices via multiple testing of correlations
• Distributions associated with simultaneous multiple hypothesis testing
• Classes of multiple decision functions strongly controlling FWER and FDR
• Comment
• Stepdown Procedures Controlling a Generalized False Discovery Rate
• Statistical nonparametric mapping: multivariate permutation tests for location, correlation, and regression problems in neuroimaging
• A note on control of the false discovery proportion
• Bootstrap Joint Prediction Regions
• A sufficient criterion for control of some generalized error rates in multiple testing
• Statistical procedures for the market graph construction
• A central limit theorem for the Benjamini-Hochberg false discovery proportion under a factor model
• StepwiseTest
• Procedures controlling the k-FDR using bivariate distributions of the null p-values
• Only closed testing procedures are admissible for controlling false discovery proportions
• An Empirical Bayes Approach to Controlling the False Discovery Exceedance
• A modified Benjamini-Hochberg multiple comparisons procedure for controlling the false discovery rate
• New procedures controlling the false discovery proportion via Romano-Wolf's heuristic
• Simultaneous critical values for t-tests in very high dimensions
• Confidence and discoveries with e-values
• Controlling the false discovery exceedance for heterogeneous tests
• Outcome-wide longitudinal designs for causal inference: a new template for empirical studies
• A Note on Controlling the Number of False Positives
• Multiple Comparisons Controlling Expected Number of False Discoveries
• Type I error rate control for testing many hypotheses: a survey with proofs
• Exceedance Control of the False Discovery Proportion
• Control of Directional Errors in Fixed Sequence Multiple Testing
• Step-up and step-down procedures controlling the number and proportion of false positives
• Balanced control of generalized error rates
• Coarse-to-fine multiple testing strategies
• FORMALIZED DATA SNOOPING BASED ON GENERALIZED ERROR RATES
• Threshold Selection in Feature Screening for Error Rate Control
• Permutation-based simultaneous confidence bounds for the false discovery proportion
• Fast closed testing for exchangeable local tests
• False Discovery Control for Multiple Tests of Association Under General Dependence
• k-FWER control without \(p \)-value adjustment, with application to detection of genetic determinants of multiple sclerosis in Italian twins
• Rejoinder on: Control of the false discovery rate under dependence using the bootstrap and subsampling
• Semi-supervised multiple testing
• A comment on: ``Invidious comparisons: ranking and selection as compound decisions by Jiaying Gu and Roger Koenker
• Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing
• Evaluations of FWER-controlling methods in multiple hypothesis testing
• Stepup procedures for control of generalizations of the familywise error rate
• Controlling individual and experimentwise error rates in replicated regular two-level factorial experiments

Describes a project that uses

Uses Software

• boot

This page was built for publication: Control of generalized error rates in multiple testing

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q155664)