Replication, statistical consistency, and publication bias (Q2437263)

The paper starts with exposing recurrent flaws in psychological science that lead some critics to announce a scientific crisis. Some researchers, either fraudulently or by ignorance, behave in such a way as to invalidate their conclusions. The most common practices include withholding null results and only reporting conclusive experiments, selecting the size of the sample while running the experiment, and multiple testing. The pressure to publish (or perish) and the need for ``sexy papers'' in order to convince leading editors encourage these practices. The author then details how one can, a posteriori, test a set of experimental results for consistency within the framework of null hypothesis significance test (NHST). The basic idea is that even if a conclusion is true, one should expect to get null findings within a set of experiments. The number of null findings one should expect in a series of NHST depends on the power or the tests used, which may be assessed from the data. The author describes the main characteristics of the resulting consistency test and reminds the reader that many papers published in leading journals failed to pass the consistency test. Last, he replies to a series of criticisms he faced after publishing evidence that many often-cited papers, including a claimed proof of precognition by Bem, Piff's experimental study associating wealth with poor moral behavior, and a few more.