Rejection odds and rejection ratios: a proposal for statistical practice in testing hypotheses

DOI10.1016/J.JMP.2015.12.007zbMATH Open1357.62018arXiv1512.08552OpenAlexW3124008874WikidataQ65002550 ScholiaQ65002550MaRDI QIDQ296932FDOQ296932

Authors: M. J. Bayarri, Daniel J. Benjamin, Thomas M. Sellke, James O. Berger 

Publication date: 24 June 2016

Published in: Journal of Mathematical Psychology (Search for Journal in Brave)

Abstract: Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 years. We propose, as an alternative, the use of the odds of a correct rejection of the null hypothesis to incorrect rejection. Both pre-experimental versions (involving the power and Type I error) and post-experimental versions (depending on the actual data) are considered. Implementations are provided that range from depending only on the p-value to consideration of full Bayesian analysis. A surprise is that all implementations -- even the full Bayesian analysis -- have complete frequentist justification. Versions of our proposal can be implemented that require only minor modifications to existing practices yet overcome some of their most severe shortcomings.

Full work available at URL: https://arxiv.org/abs/1512.08552

Recommendations

• Bayesian hypothesis testing: redux
• Revised standards for statistical evidence
• The use of \(p\)-values in applied research: interpretation and new trends
• Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence
• Adaptative significance levels using optimal decision rules: balancing by weighting the error probabilities

zbMATH Keywords

BayesianfrequentistBayes factorsodds

Mathematics Subject Classification ID

Parametric hypothesis testing (62F03) Bayesian inference (62F15) Foundations and philosophical topics in statistics (62A01) Sequential statistical analysis (62L10) Applications of statistics to psychology (62P15)

Cites Work

• Title not available (Why is that?)
• Calibration of \(\rho \) values for testing precise null hypotheses
• Title not available (Why is that?)
• Statistical decision theory and Bayesian analysis. 2nd ed
• Default Bayes Factors for Nonnested Hypothesis Testing
• Could Fisher, Jeffreys and Neyman have agreed on testing? (With comments and a rejoinder).
• Frequentist probability and frequentist statistics
• Unified frequentist and Bayesian testing of a precise hypothesis. With comments by Dennis V. Lindley, Thomas A. Louis and David Hinkley and a rejoinder by the authors
• A unified conditional frequentist and Bayesian test for fixed and sequential simple hypothesis testing
• Revised standards for statistical evidence
• Smoothness and Convex Area Functionals—Revisited
• Title not available (Why is that?)
• Unified Conditional Frequentist and Bayesian Testing of Composite Hypotheses
• Fixed-Sample-Size Analysis of Sequential Observations

Cited In (23)

• Inference for negativist theory using numerically computed rejection regions
• Multiple Improvements of Multiple Imputation Likelihood Ratio Tests
• Replication success under questionable research practices -- a simulation study
• Bayesian Calibration of p‐Values from Fisher's Exact Test
• Statistical paradises and paradoxes in big data. I: Law of large populations, big data paradox, and the 2016 US presidential election
• Benjamin, D. J., and Berger, J. O. (2019), “Three Recommendations for Improving the Use of p-Values”, The American Statistician, 73, 186–191: Comment by Foulley
• Optional stopping with Bayes factors: a categorization and extension of folklore results, with an application to invariant situations
• How the Maximal Evidence of P-Values Against Point Null Hypotheses Depends on Sample Size
• The use of p-values in applied research: Interpretation and new trends
• Editors' introduction to the special issue ``Bayes factors for testing hypotheses in psychological research: practical relevance and new developments
• Neyman–Pearson lemma for Bayes factors
• Null Hypothesis Significance Testing Interpreted and Calibrated by Estimating Probabilities of Sign Errors: A Bayes-Frequentist Continuum
• Time to dispense with the \(p\)-value in OR? Rationale and implications of the statement of the American Statistical Association (ASA) on \(p\)-values
• A Bayes Factor for Replications of ANOVA Results
• Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies
• A note on Type S/M errors in hypothesis testing
• A tutorial on bridge sampling
• Title not available (Why is that?)
• Putting the P-Value in its Place
• An Introduction to Second-Generation p-Values
• The impact of credit on economic growth in Vietnam: a comparison of traditional methods and the Bayes method
• Bayesian model selection in the \(\mathcal{M}\)-open setting -- approximate posterior inference and subsampling for efficient large-scale leave-one-out cross-validation via the difference estimator
• Three Recommendations for Improving the Use of p-Values

This page was built for publication: Rejection odds and rejection ratios: a proposal for statistical practice in testing hypotheses

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q296932)