A Recap of the Tilburg Meta-Research Day
On Friday November 22, 2019, the Meta-Research Center at Tilburg University organized the Tilburg Meta-Research Day. Around 90 interested researchers attended this day that involved three plenary lectures, by John Ioannidis (who received an honorary doctorate from Tilburg University a day earlier), Ana Marušić, and Sarah de Rijcke, and seven parallel sessions on meta-research.
Below you can find the links to the video footage of the three plenary sessions as well as summaries of all seven parallel sessions. The full program of the Tilburg Meta-Research Day can be found here. If you have any questions or comments, please contact us at metaresearch@uvt.nl.
Next up at Tilburg: The 1st European Conference on Meta-Research (July 2021).
Recordings of plenary talks:
Plenary talk by Sarah de Rijcke: Research on Research Evaluation: State-of-the-art and practical insights
Plenary talk by Ana Marušić: Reviewing Reviews: Research on the Review Process at Journals and Funding Agencies
Plenary talk by John Ioannidis: Meta-research in different scientific fields: What lessons can we learn from each other?
Parallel sessions (see below for summaries):
How can meta-research improve research evaluation? (Session leaders: Sarah de Rijcke & Rinze Benedictus)
How can we ensure the future of meta-research? (Session leader: Olmo van den Akker)
How can meta-research improve statistical practices? (Session leader: Judith ter Schure)
How can meta-research improve the Psychological Science Accelerator (PSA) and how can the PSA improve meta-research? (Session leaders: Peder Isager & Marcel van Assen)
How can meta-research improve peer review? (Session leader: Ana Marušić)
How can meta-research improve our understanding of the effects of incentives on the efficiency and reliability of science? (Session leaders: Sophia Crüwell, Leonid Tiokhin, & Maia Salholz-Hillel)
Many Paths: A new way to communicate, discuss, and conduct (meta-)research (Session leaders: Hans van Dijk & Esther Maassen)
How can meta-research improve research evaluation
Session leaders: Sarah de Rijcke & Rinze Benedictus
The evaluation of research and researchers is currently based on biased metrics like the H-index and the journal impact factor. Several new initiatives have been launched in favor of indicators that correspond better to actual research quality. One of these initiatives is “Redefine excellence” from the University Medical Center (UMC) Utrecht. In this session, Rinze Benedictus shortly outlined the innovations that are implemented at the UMC Utrecht, after which Sarah de Rijcke led a discussion on how we can properly evaluate whether these innovations are effective.
The session stimulated a productive discussion about differences and similarities between sociology of science and meta-research. Both fields could be termed ‘research on research’, but they appear to be rather distinct, using very different languages, concepts and maybe even springing from different concerns. However, the feeling in the session was that a lot could be gained by more interaction between the fields.
Promising ways to build bridges seem:
Shared conferences to share concepts, language and maybe even research questions. A thematic approach (as opposed to method-based) to research questions could also facilitate interaction.
Identification of stakeholders: why are we doing research? For who?
Shared teaching, e.g. through setting up a joint workshop by CWTS and Tilburg University/Department of Methodology
How can we ensure the future of meta-research?
Session leader: Olmo van den Akker
In this session, we set out to identify how we can ensure that the field of meta-research will remain vital in the upcoming years. Although the original focus of the session was to identify grant opportunities for meta-research projects, the discussion quickly developed into identifying journals that are open to submissions of meta-research studies. We aimed to draft a list of such journals, which can be found here. The list is far from exhaustive so please add journals if you can. The list mainly pertains to journals and journal collections specifically catered to meta-research, but there are of course also general journals that welcome meta-research submissions. In that sense, we are lucky as meta-researchers that our studies are often suitable for a wide variety of different journals.
That being said, one sentiment that arose in our discussion is that we still feel that we are missing a broad meta-research journal purely for meta-research papers. Such a journal would increase the visibility of our field, but there’s also the danger that more substantive researchers would engage less with meta-research studies published in a journal like this (as opposed to journals in their substantive field). However, we concluded that this might not be so problematic given that the majority of researchers use Google Scholar or other databases to look for papers and are less and less committed to only reading their papers from a few of their favorite journals. Below you can find a list of things that we thought would be valuable to consider when launching a specific meta-research journal.
The journal should be broad and welcome submissions from all areas of meta-research (and even meta-meta-research). As long as studying the process and outcomes of science is critical.
It would be good to have the journal link meta-research to the philosophy of science and science and technology studies (STS) as it appears that these related fields currently do not work together as much as they could.
It would be great if this journal would incorporate the latest meta-research on the effectiveness of journals as journal policy.
The journal could even be a trial ground for journal innovations. For example, the journal could try out whether a designated statistical reviewer for each submission would work (like is customary in medicine) or try out technological innovations facilitating SMART preregistration, multiverse analyses.
Initiating a Meta-Research Society with a dedicated conference could help fund the journal through society fees and conference fees.
The journal would do well to implement the CRediT authorship guidelines.
Preregistration, open data, open code, and open materials should be required, unless authors can convince the editorial team that it is not necessary in their case.
The editorial board should be paid, because a committed editorial board is crucial for the longevity and credibility of the journal. Preferably also reviewers would be paid, but this would require substantially more funding.
In the summer of 2021 Tilburg University will organize another Meta-Research Conference, this one will probably consist of two days and will focus more on the dissemination of meta-research studies. This conference could be a great place to launch a meta-research society and an accompanying meta-research journal.
How can meta-research improve statistics?
Session leader: Judith ter Schure
How can meta-research improve statistics? The conclusion we reached is that it varies a lot per field whether scientists in their experimental design actually feel like they contribute to an accumulating series of studies. In some fields awareness exists that the results of an experiment will someday end up in a meta-analysis with existing experiments, while in others scientists aim to design experiments as 'refreshingly new' as possible. In a table that shows series of studies together in one column if they could be meta-analyzed, this latter approach shows scientists who mainly aim to initiate new columns. This pre-experimental perspective might be different from the meta-analysis perspective, in which a systematic search and inclusion criteria might still force those experiments together in one column, even though they weren't intended that way. This practice might erode trust in meta-analyses that try to synthesize effects from too different experiments.
The discussion was very hesitant towards enforcing rules (e.g. by funders or universities) on scientists in priority setting, such as whether a field needs more columns of 'refreshingly new' experiments, or needs replications of existing studies (extra rows) so a field can settle on a specific topic in one column with a meta-analysis.
In terms of statistical consequences, sequential processes might still be at play if scientists designing experiments know about the results of other experiments that might end up in the same meta-analysis. Full exchangeability in meta-analysis means that no-one would have decided differently on the feasibility or design of an experiment had the results of others been different. If that assumption cannot be met, we should consider studies as part of series in our statistical meta-analysis, even without forcing this approach in the design phase.
Meta-research and the Psychological Science Accelerator
Session leaders: Marcel van Assen & Peder Isager
The Psychological Science Accelerator (PSA) is a standing network of more than 500 laboratories that collect large-scale, non-WEIRD data for psychology studies (see https://psysciacc.org and https://osf.io/93qpg). The PSA is currently running six many-lab projects, and a number of proposed future projects are currently under review. Importantly, the PSA has established a meta-research working group that is currently examining both how the PSA can best interface with the meta-research community, and how meta-research can help bolster the quality of research projects conducted at the PSA (see https://docs.google.com/document/d/1D-NmvFE4qaC-dXAWQn16SBLsY9AABCrm8jDDy3-cD8w/edit?usp=sharing)
The session began with an overview of PSA’s organization, presented by Peder, and a discussion of the importance of many-lab studies, presented by Marcel. The slides for these presentations can be found at https://osf.io/wnyga. Afterwards, the majority of the session was devoted to discussing seven predetermined topics related to how the meta-research field and the PSA may learn from each other. Participants could either independently provide their suggestions on the seven topics in a google doc (https://bit.ly/2KIUHTW) or on paper. After about half an hour independently working on the topics, we discussed the participants’ suggestions in the remainder of the session.
The following conclusions can be drawn from our discussion:
There are multiple ways in which the PSA could contribute to meta-research (e.g. by providing access to lab data and project-level data for conducted studies, and by allowing researchers to vary properties of research designs - like the measurement tools - to study effect size heterogeneity, and advance theory by examining boundary conditions).
There are multiple issues within the meta-research field that seems relevant to the PSA. Issues related to theory, measurement and sample size determination were emphasized in particular.
Meta-researchers seem interested in contributing to the PSA research endeavor, but emphasize a lack of both general information about the PSA organization and specific information about what contributions could/would entail (e.g. what volunteer efforts one could contribute to and what studies would be relevant for the “piggy-back” submission policy).
In summary, there seems to be much enthusiasm for the PSA within the meta-research community, and there are many overlapping interests between the PSA and the meta-research community. The points raised in this session will be communicated to the PSA network of researchers, with the hope that it will help facilitate more communication between the two research communities in the future.
Other resources
PSA Data & methods committee bylaws: https://osf.io/p65qe/
Proposing a theory committee at the PSA (blog post): https://pedermisager.netlify.com/post/psa-theory-committee/
How can meta-research improve peer review?
Session leader: Ana Marušić
The session started with a discussion about research approaches to different types of peer review: single blind, double blind, consultative, results free, open, and post-publication peer review. In post-publication peer review, the system that was pioneered by the F1000 Research, peer review is completely open to study, as all steps in the peer review process and editorial decision making are transparent and available in the public domain. This is not possible for other types of peer review, which remain elusive to researchers. Even in journals that publish the prepublication history of an article (like BioMed Central journals in biomedicine), the information on the review process is available only for published articles, but not for those that were rejected (and which represent the majority of articles submitted to a journal). This is a serious hindrance to meta-research on journal peer review.
The participants discussed the possibilities of having access to complete peer review data, and the recent activities by the COST Action PEERE – New Frontiers in Peer Review, were discussed. PEERE brought together the researchers and publishers to establish a database on peer review in journals from different disciplines in order to study all aspects of peer review.
The participants in the session also discussed differences in peer review in different disciplines, as well as the need for qualitative studies on peer review. This methodological approach would be particularly important in understanding preferences and habits of peer reviewers. Recent findings, both from surveys and analysis of peer review in journals show that researchers prefer double blind peer review when they are invited to review for a journal. A qualitative approach would be useful to understand this phenomenon and build hypotheses for testing in a quantitative methodological approach.
How can meta-Research improve research incentives?
Session leaders: Sophia Crüwell, Leo Tiokhin, & Maia Salholz-HillelEveryone’s talking about “the incentives,” but what does that mean? How can we move beyond our intuitions and towards a deeper understanding of how incentives affect the efficiency and reliability of science? The aim of this session was to explore the role of incentives in science, with the goal of facilitating a broader discussion of what important questions remain unanswered.
Some conclusions from our discussion are outlined below. We would like to invite both session participants and the wider community to contribute to the following library of resources on (meta)research relevant to incentives in science: https://www.zotero.org/groups/2421057/incentives_in_academic_science.
Some conclusions from our discussion:
We need to split incentives, stakeholders, behaviors, and outcomes.
Should we be focusing on predictors of career success rather than on incentives? However, career success is the outcome, which incentivizes the behaviors (e.g. publications).
We need to understand the parameters within which each incentive operates, i.e., a cost-benefit assessment towards outcomes. We could create a mapping or taxonomy to move the conversation forward. We could do this through an iterative, cross-stakeholder process that would then allow us to decide on next steps.
Rational choice theory
Delphi method: cyclical process for circulating solutions between
We should consider both intrinsic and extrinsic incentives.
Intrinsic incentives include what a person values, such as a desire to help patients, discover something about the world, etc. Extrinsic incentives include tenure and other career payoffs, prestige, etc. The external may crowd out internal incentives.
Is it possible to separate them? For example, proximate/ultimate from biology. However, intrinsic vs. extrinsic may be a false dichotomy. Extrinsic incentives shape intrinsic ones.
From a Mertonian sociology of science perspective, the drive to make a discovery is as strong a drive to refute a discovery. But this doesn’t seem to be the case. So, what are researchers trying to optimize?
Why do incentives exist? They are used as a proxy to measure who is a good scientist. E.g., measured by papers, publication, citation.
Why do people leave science?
Possible definitions of incentives
An ontology/framework of types of incentives & what questions you should ask about them; is it a positive or negative incentive?
Approach & avoidance approach
Incentive can also be the purpose
Lots of theories of behavior change already exist; do we need to reinvent the wheel?
Should we be talking about specific incentives?
Do incentivized behaviors have to be intentional?
Knowledge deficiency approach
Many Paths: A new way to conduct, discuss, and communicate (meta-)research
Session leaders: Hans van Dijk & Esther Maassen (in collaboration with Liberate Science)
Slides: https://github.com/emaassen/talks/blob/master/191122-mrd-many-paths.pdf
In Many Paths, we invite researchers from multiple disciplines to participate in a collaborative project to answer one research question, and we allow an emergent process to occur in the theory, data, results, and conclusion steps thereafter. Given that results are often path dependent, and *many paths* can be taken in a research process, we aim to examine what paths a research project initiates, prunes, and merges. The Many Paths model offers insight into how researchers from different disciplines approach and study the same question. We conduct and communicate the Many Paths research process in steps ("as-you-go"), instead of after the research is completed ("after-the-fact"). During our session, we also discussed the relationship of Many Paths to previous Many Projects (i.e., the Reproducibility Project Psychology, Many Labs, and Many Analysts).
Our goal of the session was to introduce the Many Paths model and to gather feedback and suggestions on the project. Reactions to the proposed model and the new way of communication were generally positive. Many Paths appears to provide the opportunity to gather a large amount of data from various disciplines in a transparent manner. It also allows for diversity and inclusivity. It would be interesting to find out if and how researchers decide to collaborate across disciplines. However, they might be hesitant to do so because of the notable difference in what they are used to now (i.e., competition) compared to what they could do (i.e., collaboration). Whereas some people claimed a project such as Many Paths would provide clear answers to the proposed research question, some expressed concerns about the possibility of excessive fragmentation or disintegration of paths, and difficulties with combining information from various conclusions and paths. Another possible issue that was mentioned relates to the quality assurance for the research output of Many Paths; a threshold should be in place to ensure contributions adhere to a certain quality. It should also be clear how the code of conduct would be enforced.
