Abstract

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies.

Keywords

Psychological scienceFalse positive paradoxNull hypothesisPsychologyMeta-analysisFalse positives and false negativesPublication biasStatistical hypothesis testingSample (material)Null (SQL)Cognitive psychologyApplied psychologySocial psychologyComputer scienceEconometricsStatisticsData miningArtificial intelligenceMedicineMathematics

Affiliated Institutions

Related Publications

Publication Info

Year
2012
Type
article
Volume
7
Issue
6
Pages
543-554
Citations
837
Access
Closed

Social Impact

Social media, news, blog, policy document mentions

Citation Metrics

837
OpenAlex
40
Influential
534
CrossRef

Cite This

Marjan Bakker, Annette van Dijk, Jelte M. Wicherts (2012). The Rules of the Game Called Psychological Science. Perspectives on Psychological Science , 7 (6) , 543-554. https://doi.org/10.1177/1745691612459060

Identifiers

DOI
10.1177/1745691612459060
PMID
26168111

Data Quality

Data completeness: 81%