More powerful multiple testing in randomized experiments with non-compliance

DOI10.5705/SS.202016.0116MaRDI QIDQ5278109zbMATH OpenOpenAlexFDO

Authors Joseph J. Lee, Laura Forastiere, Natesh S. Pillai, Luke W. Miratrix

Publication date 13 July 2017

Published in STATISTICA SINICA (Search for Journal in Brave)

Full work available at URL https://arxiv.org/abs/1605.07242

zbMATH Keywords

causal inferencehypothesis testingmultiple comparisonsrandomization-based inferenceposterior predictive \(p\)-valueprincipal stratifiation

Mathematics Subject Classification ID

Applications of statistics to economics (62P20) Paired and multiple comparisons; multiple testing (62J15)

Abstract: Two common concerns raised in analyses of randomized experiments are (i) appropriately handling issues of non-compliance, and (ii) appropriately adjusting for multiple tests (e.g., on multiple outcomes or subgroups). Although simple intention-to-treat (ITT) and Bonferroni methods are valid in terms of type I error, they can each lead to a substantial loss of power; when employing both simultaneously, the total loss may be severe. Alternatives exist to address each concern. Here we propose an analysis method for experiments involving both features that merges posterior predictive $p$-values for complier causal effects with randomization-based multiple comparisons adjustments; the results are valid familywise tests that are doubly advantageous: more powerful than both those based on standard ITT statistics and those using traditional multiple comparison adjustments. The operating characteristics and advantages of our method are demonstrated through a series of simulated experiments and an analysis of the United States Job Training Partnership Act (JTPA) Study, where our methods lead to different conclusions regarding the significance of estimated JTPA effects.

Recommendations

• Posterior predictive \(p\)-values with Fisher randomization tests in noncompliance settings: test statistics vs discrepancy measures
• Bayesian inference for causal effects in randomized experiments with noncompliance
• On regression adjustments in experiments with several treatments
• Stratified Analysis in Randomized Trials with Noncompliance
• A simple test for the ignorability of non-compliance in experiments

Cited in

(4)

• Posterior predictive \(p\)-values with Fisher randomization tests in noncompliance settings: test statistics vs discrepancy measures
• scientific article; zbMATH DE number 2147398 (Why is no real title available?)
• Valid randomization-based p-values for partially post hoc subgroup analyses
• A simple test for the ignorability of non-compliance in experiments

This page was built for publication: More powerful multiple testing in randomized experiments with non-compliance

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q5278109)