Beyond the two-trials rule: Type-I error control and sample size planning with the sceptical p-value

DOI10.48550/ARXIV.2207.00464arXiv2207.00464OpenAlexW4381379033MaRDI QIDQ65586FDOQ65586

Leonhard Held, Charlotte Micheloud, Charlotte Micheloud, Leonhard Held, Fadoua Balabdaoui, Fadoua Balabdaoui

Publication date: 1 July 2022

Published in: Statistica Neerlandica (Search for Journal in Brave)

Abstract: We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical $p$-value. A recalibration is proposed to obtain exact overall Type-I error control if the effect is null in both studies and additional bounds on the partial and conditional Type-I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two-trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two-trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.

Full work available at URL: https://arxiv.org/abs/2207.00464

zbMATH Keywords

design of replication studiespower calculationsreplicabilitysceptical \(p\)-valuetwo-trials ruleType-I error control

Mathematics Subject Classification ID

Statistics (62-XX) Applications of statistics (62Pxx) Parametric inference (62Fxx)

Cites Work

• Power calculations for replication studies
• Probability and random processes.
• An unexpected encounter with Cauchy and Lévy
• Sampling and Bayes' Inference in Scientific Modelling and Robustness
• “Last-Place Aversion”: Evidence and Redistributive Implications *
• Statistical Issues in Drug Development
• Introduction to Randomized Controlled Clinical Trials
• The p-value Function and Statistical Inference
• The assessment of replication success based on relative effect size

Cited In (4)

• P-Value Precision and Reproducibility
• Beyond the two-trials rule
• How redefining statistical significance can worsen the replication crisis
• Replication study design: confidence intervals and commentary

This page was built for publication: Beyond the two-trials rule: Type-I error control and sample size planning with the sceptical $p$-value

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q65586)