Multiple testing for exploratory research

DOI10.1214/11-STS356zbMATH Open1331.62369arXiv1208.2841OpenAlexW3099580112MaRDI QIDQ5963767FDOQ5963767

Jelle J. Goeman, Aldo Solari

Publication date: 23 February 2016

Published in: Statistical Science (Search for Journal in Brave)

Abstract: Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing.

Full work available at URL: https://arxiv.org/abs/1208.2841

zbMATH Keywords

false discovery proportionconfidence setclosed testing

Mathematics Subject Classification ID

Parametric hypothesis testing (62F03) Paired and multiple comparisons; multiple testing (62J15)

Cites Work

• Hierarchical testing of variable importance
• Title not available (Why is that?)
• An improved Bonferroni procedure for multiple tests of significance
• The control of the false discovery rate in multiple testing under dependency.
• On closed testing procedures with special reference to ordered analysis of variance
• A Direct Approach to False Discovery Rates
• Shortcuts for Locally Consonant Closed Test Procedures
• Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons
• Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
• Empirical Bayes Analysis of a Microarray Experiment
• Control of generalized error rates in multiple testing
• Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures
• Screening for Partial Conjunction Hypotheses
• Title not available (Why is that?)
• Estimating the Proportion of True Null Hypotheses, with application to DNA Microarray Data
• False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters
• Some probability inequalities for ordered MTP\(_2\) random variables: a proof of the Simes conjecture
• On the False Discovery Rate and Expected Type I Errors
• False Discovery Control for Multiple Tests of Association Under General Dependence
• Title not available (Why is that?)
• The sequential rejection principle of familywise error control
• Optimal testing of multiple hypotheses with common effect direction
• Closed procedures are better and often admit a shortcut.
• Hochberg's Step-Up Method: Cutting Corners Off Holm's Step-Down Method

Cited In (37)

• A tutorial on statistically sound pattern discovery
• Hierarchical inference for genome-wide association studies: a view on methodology with software
• Estimating minimum effect with outlier selection
• Hommel's procedure in linear time
• Replicability across multiple studies
• Admissible ways of merging \(p\)-values under arbitrary dependence
• Simultaneous high-probability bounds on the false discovery proportion in structured, regression and online settings
• Adaptive novelty detection with false discovery rate guarantee
• A multiple testing method for hypotheses structured in a directed acyclic graph
• Post hoc confidence bounds on false positives using reference families
• Large‐scale simultaneous inference under dependence
• Post-selection inference for e-value based confidence intervals
• Exact testing with random permutations
• Closed Testing with Globaltest, with Application in Metabolomics
• Title not available (Why is that?)
• Combining partial true discovery guarantee procedures
• Permutation-based true discovery proportions for functional magnetic resonance imaging cluster analysis
• Mediation effect selection in high-dimensional and compositional microbiome data
• Only closed testing procedures are admissible for controlling false discovery proportions
• Testing with \(\mathrm{p}^*\)-values: between p-values, mid p-values, and e-values
• True and false discoveries with independent and sequential \(e\)-values
• Confidence and discoveries with \(e\)-values
• Trade-off Between Validity and Efficiency of Merging P-Values Under Arbitrary Dependence
• Controlling the false discovery exceedance for heterogeneous tests
• Testing partial conjunction hypotheses under dependency, with applications to meta-analysis
• F-distribution calibrated empirical likelihood ratio tests for multiple hypothesis testing
• False Discovery Proportion Estimation by Permutations: Confidence for Significance Analysis of Microarrays
• A region-based multiple testing method for hypotheses ordered in space or time
• Filtering the Rejection Set While Preserving False Discovery Rate Control
• Logical and test consistency in pairwise multiple comparisons
• E-values: calibration, combination and applications
• Exceedance control of the false discovery proportion via high precision inversion method of berk-Jones statistics
• Semi-supervised multiple testing
• Controlling costs: feature selection on a budget
• The online closure principle
• Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari
• Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari

   Recommendations

• Correction note: ``Multiple testing for exploratory research 👍 👎
• Title not available (Why is that?) 👍 👎
• Title not available (Why is that?) 👍 👎
• Multistage tests of multiple hypotheses 👍 👎
• Multiple testing in a set of nested hypotheses 👍 👎
• Multiple Comparisons 👍 👎
• Multiple testing of causal hypotheses 👍 👎
• Multivariate Multiple Test Procedures 👍 👎
• Multiple Testing for Dose Finding 👍 👎

This page was built for publication: Multiple testing for exploratory research

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5963767)