Abstract: Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing.
Recommendations
- Correction note: ``Multiple testing for exploratory research
- scientific article; zbMATH DE number 3949560
- scientific article; zbMATH DE number 3905661
- Multistage tests of multiple hypotheses
- Multiple testing in a set of nested hypotheses
- Multiple Comparisons
- Multiple testing of causal hypotheses
- Multivariate Multiple Test Procedures
- Multiple testing for dose finding
Cites work
- scientific article; zbMATH DE number 3831073 (Why is no real title available?)
- scientific article; zbMATH DE number 720689 (Why is no real title available?)
- scientific article; zbMATH DE number 2220287 (Why is no real title available?)
- A Direct Approach to False Discovery Rates
- An improved Bonferroni procedure for multiple tests of significance
- Closed procedures are better and often admit a shortcut.
- Control of generalized error rates in multiple testing
- Empirical Bayes Analysis of a Microarray Experiment
- Estimating the Null and the Proportion of Nonnull Effects in Large-Scale Multiple Comparisons
- Estimating the Proportion of True Null Hypotheses, with application to DNA Microarray Data
- False Discovery Control for Multiple Tests of Association Under General Dependence
- False Discovery Rate–Adjusted Multiple Confidence Intervals for Selected Parameters
- Hierarchical testing of variable importance
- Hochberg's Step-Up Method: Cutting Corners Off Holm's Step-Down Method
- Lower bounds for the number of false null hypotheses for multiple testing of associations under general dependence structures
- On closed testing procedures with special reference to ordered analysis of variance
- On the False Discovery Rate and Expected Type I Errors
- Optimal testing of multiple hypotheses with common effect direction
- Rectangular Confidence Regions for the Means of Multivariate Normal Distributions
- Screening for Partial Conjunction Hypotheses
- Shortcuts for locally consonant closed test procedures
- Some probability inequalities for ordered MTP₂ random variables: a proof of the Simes conjecture
- The control of the false discovery rate in multiple testing under dependency.
- The sequential rejection principle of familywise error control
Cited in
(78)- Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari
- Discussion of ``Multiple testing for exploratory research by J. J. Goeman and A. Solari
- A tutorial on statistically sound pattern discovery
- On agnostic post-hoc approaches to false positive control
- Hierarchical inference for genome-wide association studies: a view on methodology with software
- Estimating minimum effect with outlier selection
- Hommel's procedure in linear time
- \(F\)-distribution calibrated empirical likelihood ratio tests for multiple hypothesis testing
- Admissible ways of merging \(p\)-values under arbitrary dependence
- Simultaneous high-probability bounds on the false discovery proportion in structured, regression and online settings
- Replicability across multiple studies
- Inverse set estimation and inversion of simultaneous confidence intervals
- FDP control in mass-univariate linear models using the residual bootstrap
- A multiple testing method for hypotheses structured in a directed acyclic graph
- Post hoc confidence bounds on false positives using reference families
- Adaptive novelty detection with false discovery rate guarantee
- Large‐scale simultaneous inference under dependence
- Controlled Discovery and Localization of Signals via Bayesian Linear Programming
- False discovery proportion estimation by permutations: confidence for significance analysis of microarrays
- Confidence bounds for the true discovery proportion based on the exact distribution of the number of rejections
- Flexible control of the median of the false discovery proportion
- Regularizing stock return covariance matrices via multiple testing of correlations
- Post-selection inference for e-value based confidence intervals
- Post-selection inference in multiverse analysis (PIMA): an inferential framework based on the sign flipping score test
- Exact testing with random permutations
- Closed Testing with Globaltest, with Application in Metabolomics
- Author's reply to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Priyantha Wijayatunga's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Ruodu Wang's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Paul Vos's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Judith Ter Schure's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Stephen Senn's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Jorge Mateu's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Ryan Martin's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Nick Longford's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Tze Leung Lai and Anna Choi's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Kuldeep Kumar's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Chloe Krakauer and Kenneth Rice's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Sander Greenland's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Christine P. Chai's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- A distillation of the live chat during the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Christian Hennig's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Vladimir Vovk's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Arthur Paul Pedersen's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Xiao-Li Meng's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Peter D. Grünwald's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Aaditya Ramdas's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Barbara Osimani's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Harry Crane's contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication' by Glenn Shafer
- Seconder of the vote of thanks to Glenn Shafer and contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication'
- Proposer of the vote of thanks to Glenn Shafer and contribution to the discussion of `Testing by betting: a strategy for statistical and scientific communication'
- Testing by betting: a strategy for statistical and scientific communication
- scientific article; zbMATH DE number 817577 (Why is no real title available?)
- Only closed testing procedures are admissible for controlling false discovery proportions
- Combining partial true discovery guarantee procedures
- Permutation-based true discovery proportions for functional magnetic resonance imaging cluster analysis
- Mediation effect selection in high-dimensional and compositional microbiome data
- Multiple testing of mix-and-match feature sets in multi-omics
- Testing with \(\mathrm{p}^*\)-values: between p-values, mid p-values, and e-values
- True and false discoveries with independent and sequential e-values
- Exploratory inference: localizing relevant effects with confidence
- Confidence and discoveries with e-values
- Trade-off Between Validity and Efficiency of Merging P-Values Under Arbitrary Dependence
- Permutation-based multiple testing when fitting many generalized linear models
- Controlling the false discovery exceedance for heterogeneous tests
- Testing partial conjunction hypotheses under dependency, with applications to meta-analysis
- A region-based multiple testing method for hypotheses ordered in space or time
- Catch me if you can: signal localization with knockoff e-values
- Logical and test consistency in pairwise multiple comparisons
- E-values: calibration, combination and applications
- Filtering the Rejection Set While Preserving False Discovery Rate Control
- A simultaneous confidence-bounded true discovery proportion perspective on localizing differences in smooth terms in regression models
- Exceedance control of the false discovery proportion via high precision inversion method of berk-Jones statistics
- Semi-supervised multiple testing
- Multiple hypothesis testing in genomics
- Too many, too improbable: testing joint hypotheses and closed testing shortcuts
- Controlling costs: feature selection on a budget
- The online closure principle
This page was built for publication: Multiple testing for exploratory research
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5963767)
