Replication, statistical consistency, and publication bias
The paper starts with exposing recurrent flaws in psychological science that lead some critics to announce a scientific crisis. Some researchers, either fraudulently or by ignorance, behave in such a way as to invalidate their conclusions. The most common practices include withholding null results and only reporting conclusive experiments, selecting the size of the sample while running the experiment, and multiple testing. The pressure to publish (or perish) and the need for ``sexy papers in order to convince leading editors encourage these practices. The author then details how one can, a posteriori, test a set of experimental results for consistency within the framework of null hypothesis significance test (NHST). The basic idea is that even if a conclusion is true, one should expect to get null findings within a set of experiments. The number of null findings one should expect in a series of NHST depends on the power or the tests used, which may be assessed from the data. The author describes the main characteristics of the resulting consistency test and reminds the reader that many papers published in leading journals failed to pass the consistency test. Last, he replies to a series of criticisms he faced after publishing evidence that many often-cited papers, including a claimed proof of precognition by Bem, Piff's experimental study associating wealth with poor moral behavior, and a few more.
- The consistency test does not -- and cannot -- deliver what is advertised: a comment on Francis (2013)
- On biases in assessing replicability, statistical consistency and publication bias
- Statistical methods for replicability assessment
- \(p_{\text{rep}}\): an agony in five fits
- Statistical proof? The problem of irreproducibility
- Fixed-Sample-Size Analysis of Sequential Observations
- scientific article; zbMATH DE number 4090552 (Why is no real title available?)
- scientific article; zbMATH DE number 48701 (Why is no real title available?)
- Introduction to Meta‐Analysis
- Operating Characteristics of a Rank Correlation Test for Publication Bias
- Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis
- Method in experiment: rhetoric and reality
- Statistical methods for replicability assessment
- Replication study design: confidence intervals and commentary
- An evaluation of statistical methods for aggregate patterns of replication failure
- How redefining statistical significance can worsen the replication crisis
- Distance from a distance: the robustness of psychological distance effects
- The consistency test may be too weak to be useful: its systematic application would not improve effect size estimation in meta-analyses
- It really just does not follow, comments on Francis (2013)
- On biases in assessing replicability, statistical consistency and publication bias
- The consistency test does not -- and cannot -- deliver what is advertised: a comment on Francis (2013)
- Clarifications on the application and interpretation of the test for excess significance and its extensions
- Interrogating \(p\)-values
- We should focus on the biases that matter: a reply to commentaries
- What type of Type I error? Contrasting the Neyman-Pearson and Fisherian approaches in the context of exact and direct replications
- Statistical proof? The problem of irreproducibility
- The falsificationist foundation for null hypothesis significance testing
- Scientific self-correction: the Bayesian way
- Variation and Covariation in Large-Scale Replication Projects: An Evaluation of Replicability
- \(p_{\text{rep}}\): an agony in five fits
- Playing with data -- or how to discourage questionable research practices and stimulate researchers to do things right
- Thinking about data, research methods, and statistical analyses. Comment on ``Playing with data -- or how to discourage questionable research practices and stimulate researchers to do things right
- Improving the conduct and reporting of statistical analysis in psychology. Response to the comments ``Thinking about data, research methods, and statistical analyses and ``Encourage playing with data and discourage questionable reporting practices provided in response to ``Playing with data -- or how to discourage questionable research practices and stimulate researchers to do things right
- On the impossibility of empirical controls of scientific theories-from the point of view of a psychologist
- \textsc{Tutorial}: ``With sufficient increases in X, more people will engage in the target behavior
This page was built for publication: Replication, statistical consistency, and publication bias
Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q2437263)
