Wrapping Up a Three (Seven)-Year Research Grant: Lessons Learned

This blogpost has been written by Michèle Nuijten. Michèle is an assistant professor of our research group who investigates reproducibility and replicability in psychology. Also, she is the developer of the tool “statcheck”. This tool automatically checks whether reported statistical results are internally consistent.

In late 2020, NWO awarded me a three-year Veni grant to develop a “Four-Step Robustness Check” to assess and improve the trustworthiness* of psychological science. Researchers were increasingly advocating for more replication research to assess trustworthiness, but this is costly and time-intensive. Therefore, I argued that we should focus on a more efficient and cost-effective way to assess trustworthiness, by first verifying reported results in separate steps before replicating them in a new sample:

1. Check for internal inconsistencies in reported statistics. (Statistical reporting)

2. Reanalyze the raw data following the original analysis plan. (Computational reproducibility)

3. Run justifiable alternative analyses to see if the result holds up. (Statistical robustness)

4. Replicate the study in a new sample. (Replicability)

Logically, you could argue that if a key result in a paper is an error (i.e. it cannot be reproduced from the underlying data), it has no clear bearing on theory or practice; it is essentially a meaningless, random number. This also means that it cannot be meaningfully compared to a finding in a replication study.  Would you have needed the replication to judge the trustworthiness of that particular finding? I don’t think so.

Despite the theoretical logic, some questions remained. What if there was only a small mismatch between the reported and recalculated number? What if there were errors in secondary analyses, but not in the primary result? What if there is not enough information available to even run a reanalysis? And can we use the logic of the four steps to design and test interventions to improve research?

Now, almost seven years (and a pandemic and two-and-a-half kids later) I’m finally wrapping up this project. Here is what I learned.

Lesson 1. Not everything that is “per definition true” holds up in practice

At the start of the project, I spent a lot of time finetuning the theoretical details of my four-step framework (Nuijten, 2020; Nuijten, 2022). I argued that some links between the steps were “per definition true”. For example, if in Step 1, it turns out that a reported result is inconsistent (e.g., the p-value does not match the accompanying test statistic and degrees of freedom), it is inevitable that a full reanalysis in Step 2 will also fail. After all, there is no scenario in which a correct reanalysis leads to an inconsistent p-value, right? Wrong.

In an exploration of the link between the four steps, I have been looking at statistical results in a sample of psychology papers (Nuijten et al., working paper). A sizable number of cases did not pass Step 1, but then did pass Step 2. It turned out that for many of these this had to do with rounding. Reported numbers are often rounded, and sometimes the rounded numbers do not neatly align with each other. When we recalculated these results with the full data, we were able to both reproduce the full, internally consistent result and the rounded version that in itself was not internally consistent.

For me, the main lesson was to move from theory to practice in time. As someone who is extensively trained in preregistration, I tend to want to think out every single detail of a project before I get started. This may not always be the best course of action, especially when exploring new terrain. Obtaining preliminary, empirical input would have significantly helped me fine-tune the theoretical framework. So note to self: run more pilot studies.

Lesson 2. It is hard to set up randomized experiments with academic journals

One of the most innovative aspects of my grant proposal was that I was going to set up actual randomized experiments in collaborations with academic journals. That way, I could actually test interventions to improve repeatability of research findings, rather than rely on observational findings – as is the standard in the field.

Oh sweet summer child.

Already when writing my proposal, I realized that finding journal collaborators was harder than I initially thought. I reasoned – perhaps naively – that if journals think your intervention is interesting and possibly worthwhile, they would be lining up to participate in your experiment. I quickly found out that journals have additional considerations.

Factors I did not take into account included the additional time an experiment may cost an editor (whose workload is often already off the charts), technical restrictions in the journal submission systems, ethical considerations (we couldn’t disclose the details of our intervention in an informed consent), and the conflicting interests of the publishers (who may be worried about declining submission or acceptance rates).

In the end, I did find two journals who were willing to participate: Collabra: Psychology and the Journal of Social Psychology (shout-out to the editors Don van Ravenzwaaij and John Edlund!). I wish I could include a “lesson learned” here that would make it significantly easier to get journals to collaborate in experiments, because I still believe that journals are an invaluable partner in testing interventions to improve scientific robustness. However, also judging from the similar difficulties that my colleagues experienced, many journals may simply see too many hurdles. I was very lucky and grateful to have found two journals willing to participate.

That said, factors that may have helped were 1) making use of the contacts in my network (it’s always easier to set up a collaboration with someone you know), 2) seeking out journals who have already shown a commitment to scientific rigor and innovation, 3) designing the experiment such that the workload is taken away from the editor as much as possible, and 4) learning the language of the publishers to convince them that the costs and risks to the journal would be minimal.

Lesson 3. However nicely you formatted your GANTT charts, real life will likely get in the way (but this may also bring opportunities)

Pretty much every grant proposal will include some type of GANTT chart: a nicely formatted time schedule with intermediate deliverables and milestones. I remember sweating over these timelines when working on my proposal until I got everything planned out just right.

But then. Almost immediately after I submitted my proposal, the world was hit with pretty much the biggest “unforeseen circumstance” of all: a global pandemic. The journals I had originally lined up for my experiment dropped out, partly because they needed all hands on deck to fast-track COVID-related research. A planned research visit to Stanford was cancelled and suddenly everybody was working from home with limited resources. In addition, I got pregnant with my first child, while at the same time my mother fell terminally ill and passed away shortly after my daughter was born, leading to prolonged sick leave.

Goodbye GANTT chart.

But even without global crises or major life events, things may (and will) change. When I finally returned to work and started on my proposed projects (punctuated by the birth of my son), it became clear that the four-step plan I envisioned, including an app and a journal-tested intervention, was not feasible in its original shape. The four steps did not align as nicely as I had thought and there was too much variation in psychological articles to create a one-size-fits-all approach (see also Lesson 1).

At the same time, these challenges came with opportunities. The COVID-19 pandemic led to a surge of scientific output, allowing us to directly study the relation between publishing under pressure and statistical errors (we found none; Van Aert, Nuijten, et al., 2023). Furthermore, even though I had to find two new journal collaborators, Collabra: Psychology and the Journal of Social Psychology turned out to fit the scope of the study better than the journals I had originally lined up. Finally, the overall delays meant that I was still working on this four-step check during the recent rise in easy-to-use AI tools. This has inspired a new angle for my final study, gave rise to new collaborations, and formed the basis of a new (and awarded!) grant proposal.

The biggest lesson in all of this was to refrain from putting too much importance on your GANTT charts. Accept that things will turn out differently than you expected, that you will need to adjust accordingly, and that this change of plans may lead to better outcomes and insights than you originally thought.

So what about the four-step robustness check?

So can we make use of the Four-Step Robustness Check to more efficiently assess and improve trustworthiness of psychological science? The short answer is: more research is needed.

During this project, I gained understanding of statistical errors: I found no evidence that working under pressure necessarily leads to more errors (Van Aert, Nuijten, et al., 2023), but I did find that implementing statcheck as a pre-submission check is associated with lower error rates (Nuijten & Wicherts, 2024). I am still waiting for the results of the experiment with the journals to make a more causal claim (Nuijten, van Ravenzwaaij, & Edlund, 2025).

It is still unclear if and how the four steps (statistical inconsistencies, computational reproducibility, statistical robustness, and replicability) are linked to each other in practice. Findings from a related collaboration (Abatayo et al., conditionally accepted) show no relation between statistical inconsistencies in the original study and replication success in a new sample, but in a current project (Nuijten et al., working paper), I’m still looking at this in more detail. Preliminary results seem to indicate that certain steps within computational reproducibility are linked, but a relation with replicability is unclear.

My main takeaway: passing any of the four steps adds to scientific quality, but we cannot (yet?) reliably predict replication success with quick verification checks.

Acknowledgements

I would like to thank NWO, the collaborating journals Collabra: Psychology and the Journal of Social Psychology, my co-authors, and my colleagues – both in the lab and around the world – for their invaluable help, insights, and inspiration to move this Veni project forward. I would also like to thank my partner for helping me regain my confidence in research after becoming a mother and losing my mother. I would like to thank my kids for not caring about this grant at all.

Footnotes

* I’m using the term “trustworthiness” here in the broad sense of the word: “Can I trust this result?”, “Is this result credible?”, “Can I consider this result a solid base for next actions?”.

Further reading

Theoretical framework of the four-step repeatability check

Nuijten, M. B. (2020). Efficient scientific self-correction in times of crisis. In E. Aarts, H. Fleuren, M. Sitskoorn & T. Wilthagen (Eds.), The new common (pp. 161-167). Springer. https://doi.org/10.1007/978-3-030-65355-2

Nuijten, M. B. (2022). Assessing and improving robustness of psychological research findings in four steps. In W. O’Donohue, A. Masuda, and S. Lilienfeld (Eds.), Avoiding questionable practices in applied psychology (pp. 379-400). Springer. https://doi.org/10.1007/978-3-031-04968-2_17

Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Almenberg, A. D., Fidler, F., Hilgard, J., Kline, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L., Schönbrodt, F., & Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73, 719-748. https://doi.org/10.1146/annurev-psych-020821-114157

Prevalence and predictors of statistical errors

Van Aert, R. C. M., Nuijten, M. B.*, Olsson-Collentine, A., Stoevenbelt, A. H., van den Akker, O. R., Klein, R. A., & Wicherts, J. M. (2023). Comparing the prevalence of statistical reporting inconsistencies in COVID-19 preprints and matched controls: a registered report. Royal Society Open Science, 10(8), 202326. https://doi.org/10.1098/rsos.202326
*shared first authorship

Nuijten, M. B. & Wicherts, J. M. (2024). Implementing statcheck during peer review is related to a steep decline in statistical reporting errors. Advances in Methods and Practices in Psychological Science, 7(2). https://doi.org/10.1177/25152459241258945

Testing an intervention to avoid statistical errors

Nuijten, M. B., Van Ravenzwaaij, D., & Edlund, J. E. (2025). Including the tool statcheck in peer review to increase analytical reproducibility: a randomized experiment in two psychology journals. Stage 1 Registered Report accepted at Peer Community In: Registered Reports. MetaArXiv. https://doi.org/10.31222/osf.io/gs4ft_v1

Empirically linking the steps of the four-step check

Abatayo, A. L., Achakulvisut, T., Acuna, D., [...], Nuijten, M. B., [...], Wilkinson, D. P., Wintle, B. C. & Wu, J. (conditionally accepted). Assessments of Credibility in the Social and Behavioral Sciences. Nature Communications.

Nuijten, M.B., Karahan, B., Tran, N.A., Hillen, S. W. (working paper). An exploratory evaluation of the four-step repeatability check: the relationship between reproducibility, robustness, and replication in psychology.