Our team member Marjan Bakker has just published a paper in Psychological Science, together with Chris Hartgerink, Jelte Wicherts and Han van der Maas. The abstract:
Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers’ experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.
The paper is available here (Open Access).
The Meta-Research group was well represented at the APS conference in Chicago. As a recap, we have shared all our slides. Feel free to view them and let us know if you have any questions or suggestions! Where applicable, Open Science Framework links are included, which makes the presentations citable as well as preserves them.
The Psychology of Statistics and the Statistics of Psychology
Estimating the reproducibility of psychological science: accounting for the statistical significance of the original study
Marcel van Assen (https://osf.io/58xqt/)
Flawed Intuitions about power
Marjan Bakker (https://osf.io/dztjs)
Honesty and Trust in Psychology Research
The Storybook Image of the Scientist
Why do so many researchers misreport p-values?
How do researchers fabricate data and how to detect fabrication?
Chris Hartgerink (https://osf.io/ucfpv/)
How to Deal with Publication Bias in Psychology? Illustrations and Recommendations
To be added
Estimation of effect sizes in the presence of publication bias: a comparison of meta-analysis methods
Publication Bias in IQ Research
Conducting meta-analyses based on p-values: Reservations and recommendations for applying p-uniform and p-curve
Robbie van Aert (https://osf.io/8rtmz/)