Preregistration is the practice of registering the hypotheses, methods, and/or analyses of a scientific study before it is conducted.
This can include analyzing primary data or secondary data. Clinical trial registration is similar, although it may not require the registration of a study's analysis protocol. Finally, registered reports include the peer review and in principle acceptance of a study protocol prior to data collection.
Preregistration assists in the identification and/or reduction of a variety of potentially problematic research practices, including
p-hacking
Data dredging (also known as data snooping or ''p''-hacking) is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. T ...
,
publication bias
In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance o ...
,
data dredging
Data dredging (also known as data snooping or ''p''-hacking) is the misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives. ...
, inappropriate forms of
post hoc analysis
In a scientific study, post hoc analysis (from Latin '' post hoc'', "after this") consists of statistical analyses that were specified after the data were seen. They are usually used to uncover specific differences between three or more group mea ...
, and (relatedly)
HARKing
HARKing (hypothesizing after the results are known) is an acronym coined by social psychologist Norbert Kerr that refers to the questionable research practice of “presenting a post hoc hypothesis in the introduction of a research report as if it ...
. It has recently gained prominence in the
open science community as a potential solution to some of the issues that are thought to underlie the
replication crisis
The replication crisis (also called the replicability crisis and the reproducibility crisis) is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibili ...
.
However, critics have argued that it may not be necessary when other open science practices are implemented such as Registered Reports.
Types
Standard preregistration
In the standard preregistration format, researchers prepare a research protocol document prior to conducting their research. Ideally, this document indicates the research hypotheses, sampling procedure, sample size, research design, testing conditions, stimuli, measures, data coding and aggregation method, criteria for data exclusions, and statistical analyses, including potential variations on those analyses. This preregistration document is then posted on a publicly available website such as the
Open Science Framework
The Center for Open Science is a non-profit technology organization based in Charlottesville, Virginia with a mission to "increase the openness, integrity, and reproducibility of scientific research." Brian Nosek and Jeffrey Spies founded the or ...
o
AsPredicted The preregistered study is then conducted, and a report of the study and its results is submitted for publication together with access to the (anonymised) preregistration document. This preregistration approach allows peer reviewers and subsequent readers to cross-reference the preregistration document with the published research article in order to identify (a) any “exploratory” tests that were not included in the preregistration document and (b) any suppressed tests that were included in the preregistered protocol but excluded from the final research report.
Registered reports
The registered report format requires authors to submit a description of the study methods and analyses prior to data collection. Once the method and analysis plan is vetted through Stage 1 peer review, publication of the findings is provisionally guaranteed. The associated study is then conducted, and the research report is submitted to Stage 2 peer review. Stage 2 peer review confirms that the actual research methods are consistent with the preregistered protocol and that quality thresholds are met (e.g., manipulation checks confirm the validity of the experimental manipulation). Studies that pass Stage 2 peer review are then published regardless of whether the results are confirming or disconfirming, significant or nonsignificant.
Hence, both preregistration and registered reports involve creating a time-stamped non-modifiable public record of the study and analysis plan before the data is collected. However, the study and analysis plan is only subjected to a formal
peer review
Peer review is the evaluation of work by one or more people with similar competencies as the producers of the work (peers). It functions as a form of self-regulation by qualified members of a profession within the relevant field. Peer review ...
before data collection in the case of registered reports.
Specialised preregistration
Preregistration can be used in relation to a variety of different research designs and methods, including:
* Quantitative research in psychology (
Bosnjak et al., 2021)
* Qualitative research (
Haven & Van Grootel, 2019)
* Preexisting data (
Mertens & Krypotosm, 2019;
Weston et al., 2019)
* Single case designs (
Johnson & Cook, 2019)
*Electroencephalogram research
Paul et al., 2021
*Experience sampling (
Kirtley et al., 2019)
*Exploratory research (
Dirnagl, 2020)
*Animal Research (
Bert et al., 2019)
Clinical trial registration
Clinical trial registration is the practice of documenting
clinical trial
Clinical trials are prospective biomedical or behavioral research studies on human participants designed to answer specific questions about biomedical or behavioral interventions, including new treatments (such as novel vaccines, drugs, dietar ...
s before they are performed in a clinical trials registry so as to combat
publication bias
In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance o ...
and
selective reporting
In epidemiology, reporting bias is defined as "selective revealing or suppression of information" by subjects (for example about past medical history, smoking, sexual experiences). In artificial intelligence research, the term reporting bias is u ...
. Registration of clinical trials is required in some countries and is increasingly being standardized. Some top medical journals will only publish the results of trials that have been
pre-registered.
A clinical trials registry is a platform which catalogs registered clinical trials.
ClinicalTrials.gov
ClinicalTrials.gov is a registry of clinical trials. It is run by the United States National Library of Medicine (NLM) at the National Institutes of Health, and is the largest clinical trials database, holding registrations from over 329,000 tri ...
, run by the
United States National Library of Medicine
The United States National Library of Medicine (NLM), operated by the United States federal government, is the world's largest medical library.
Located in Bethesda, Maryland, the NLM is an institute within the National Institutes of Health. Its ...
(NLM) was the first online registry for clinical trials, and remains the largest and most widely used. In addition to combating bias, clinical trial registries serve to increase transparency and access to clinical trials for the public. Clinical trials registries are often searchable (e.g. by disease/indication, drug, location, etc.). Trials are registered by the pharmaceutical, biotech or medical device company (Sponsor) or by the hospital or foundation which is sponsoring the study, or by another organization, such as a
contract research organization In the life sciences, a contract research organization (CRO) is a company that provides support to the pharmaceutical, biotechnology, and medical device industries in the form of research services outsourced on a contract basis. A CRO may provid ...
(CRO) which is running the study.
There has been a push from governments and international organizations, especially since 2005, to make clinical trial information more widely available and to standardize registries and processes of registering. The
World Health Organization
The World Health Organization (WHO) is a specialized agency of the United Nations responsible for international public health. The WHO Constitution states its main objective as "the attainment by all peoples of the highest possible level of h ...
is working toward "achieving consensus on both the minimal and the optimal operating standards for trial registration".
Creation and development
For many years, scientists and others have worried about reporting biases such that negative or null results from initiated clinical trials may be less likely to be published than positive results, thus skewing the literature and our understanding of how well interventions work.
This worry has been international and written about for over 50 years.
One of the proposals to address this potential bias was a comprehensive register of initiated clinical trials that would inform the public which trials had been started.
Ethical issues were those that seemed to interest the public most, as trialists (including those with potential commercial gain) benefited from those who enrolled in trials, but were not required to “give back,” telling the public what they had learned.
Those who were particularly concerned by the double standard were systematic reviewers, those who summarize what is known from clinical trials. If the literature is skewed, then the results of a systematic review are also likely to be skewed, possibly favoring the test intervention when in fact the accumulated data do not show this, if all data were made public.
ClinicalTrials.gov
ClinicalTrials.gov is a registry of clinical trials. It is run by the United States National Library of Medicine (NLM) at the National Institutes of Health, and is the largest clinical trials database, holding registrations from over 329,000 tri ...
was originally developed largely as a result of breast cancer consumer lobbying, which led to authorizing language in the
FDA Modernization Act of 1997
The United States Food and Drug Administration Modernization Act of 1997 (FDAMA) amended the Federal Food, Drug, and Cosmetic Act. This act is related to the regulation of food, drugs, devices, and biological products by the FDA. These changes ...
(Food and Drug Administration Modernization Act of 1997. Pub L No. 105-115, §113 Stat 2296), but the law provided neither funding nor a mechanism of enforcement. In addition, the law required that ClinicalTrials.gov only include trials of serious and life-threatening diseases.
Then, two events occurred in 2004 that increased public awareness of the problems of reporting bias. First, the then-New York State Attorney General
Eliot Spitzer
Eliot Laurence Spitzer (born June 10, 1959) is an American politician and attorney. A member of the Democratic Party (United States), Democratic Party, he was the 54th governor of New York from 2007 until his resignation in 2008.
Spitzer was b ...
sued
GlaxoSmithKline
GSK plc, formerly GlaxoSmithKline plc, is a British multinational pharmaceutical and biotechnology company with global headquarters in London, England. Established in 2000 by a merger of Glaxo Wellcome and SmithKline Beecham. GSK is the ten ...
(GSK) because they had failed to reveal results from trials showing that certain antidepressants might be harmful.
Shortly thereafter, the
International Committee of Medical Journal Editors
The ICMJE recommendations (full title, "Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals") are a set of guidelines produced by the International Committee of Medical Journal Editors for stan ...
(ICMJE) announced that their journals would not publish reports of trials unless they had been registered. The ICMJE action was probably the most important motivator for trial registration, as investigators wanted to reserve the possibility that they could publish their results in prestigious journals, should they want to.
In 2007, the
Food and Drug Administration Amendments Act of 2007
President of the United States George W. Bush signed the Food and Drug Administration Amendments Act of 2007 (FDAAA) on September 27, 2007. This law reviewed, expanded, and reaffirmed several existing pieces of legislation regulating the FDA. The ...
(FDAAA) clarified the requirements for registration and also set penalties for non-compliance (Public Law 110-85. The Food and Drug Administration Amendments Act of 2007
International participation
The
Uniform Requirements for Manuscripts Submitted to Biomedical Journals#International Committee of Medical Journal Editors, International Committee of Medical Journal Editors (ICMJE) decided that from July 1, 2005 no trials will be considered for publication unless they are included on a clinical trials registry. The
World Health Organization
The World Health Organization (WHO) is a specialized agency of the United Nations responsible for international public health. The WHO Constitution states its main objective as "the attainment by all peoples of the highest possible level of h ...
has begun the push for clinical trial registration with the initiation of the International Clinical Trials Registry Platform. There has also been action from the pharmaceutical industry, which released plans to make clinical trial data more transparent and publicly available. Released in October 2008, the revised
Declaration of Helsinki
The Declaration of Helsinki (DoH, fi, Helsingin julistus, sv, Helsingforsdeklarationen) is a set of ethical principles regarding human experimentation developed originally in 1964 for the medical community by the World Medical Association (WMA) ...
, states that "Every clinical trial must be registered in a publicly accessible database before recruitment of the first subject."
The World Health Organization maintains an international registry portal a
http://apps.who.int/trialsearch/ WHO states that the international registry's mission is "to ensure that a complete view of research is accessible to all those involved in health care decision making. This will improve research transparency and will ultimately strengthen the validity and value of the scientific evidence base."
Since 2007, the
Uniform Requirements for Manuscripts Submitted to Biomedical Journals#International Committee of Medical Journal Editors, International Committee of Medical Journal Editors ICMJE accepts all primary registries in the WHO network in addition to clinicaltrials.gov. Clinical trial registration in other registries excluding ClinicalTrials.gov has increased irrespective of study designs since 2014.
Reporting compliance
Various studies have measured the extent to which various trials are in compliance with the reporting standards of their registry.
Overview of clinical trial registries
Worldwide, there is growing number of registries. A 2013 study
identified the following top five registries (numbers updated as of August 2013):
Overview of preclinical study registries
Similar to clinical research, preregistration can help to improve transparency and quality of research data in preclinical research. In contrast to clinical research where preregistration is mandatory for vast parts it is still new in preclinical research. A large part of preclinical and basic biomedical research relies on animal experiments. The non-publication of results gained from animal experiments not only distorts the state of research by reinforcing the publication bias, it further represents an ethical issue. Preregistration is discussed as a measure that could counteract this problem. Following registries are suited for the preregistration of preclinical studies.
Journal support
Over 200 journals offer a registered reports option
Centre for Open Science, 2019,
and the number of journals that are adopting registered reports is approximately doubling each year (
Chambers et al., 2019).
''
Psychological Science
''Psychological Science'', the flagship journal of the Association for Psychological Science (APS), is a monthly, peer-reviewed, scientific journal published by SAGE Publications.
Publication scope
''Psychological Science'' publishes research r ...
'' has encouraged the preregistration of studies and the reporting of effect sizes and confidence intervals. The
editor-in-chief
An editor-in-chief (EIC), also known as lead editor or chief editor, is a publication's editorial leader who has final responsibility for its operations and policies.
The highest-ranking editor of a publication may also be titled editor, managing ...
also noted that the editorial staff will be asking for replication of studies with surprising findings from examinations using small sample sizes before allowing the manuscripts to be published.
''
Nature Human Behaviour
''Nature Human Behaviour'' is a monthly multidisciplinary online-only peer-reviewed scientific journal covering all aspects of human behaviour. It was established in January 2017 and is published by Springer Nature Publishing. The editor-in-chief ...
'' has adopted the registered report format, as it “shift
the emphasis from the results of research to the questions that guide the research and the methods used to answer them”.
''
European Journal of Personality
The ''European Journal of Personality'' (EJP) is the official bimonthly academic journal of the European Association of Personality Psychology covering research on personality, published by SAGE Publishing. According to citation reports based on ...
'' defines this format: “In a registered report, authors create a study proposal that includes theoretical and empirical background, research questions/hypotheses, and pilot data (if available). Upon submission, this proposal will then be reviewed prior to data collection, and if accepted, the paper resulting from this peer-reviewed procedure will be published, regardless of the study outcomes.”
Note that only a very small proportion of academic journals in psychology and neurosciences explicitly stated that they welcome submissions of replication studies in their aim and scope or instructions to authors. This phenomenon does not encourage the reporting or even attempt on replication studies.
Overall, the number of participating journals is increasing, as indicated by the
Center for Open Science
The Center for Open Science is a non-profit technology organization based in Charlottesville, Virginia with a mission to "increase the openness, integrity, and reproducibility of scientific research." Brian Nosek and Jeffrey Spies founded the or ...
, which maintains a list of journals encouraging the submission of registered reports.
Rationale
Several articles have outlined the rationale for preregistration (e.g.
Lakens, 2019 Nosek et al., 2018;
Wagenmakers et al., 2012).
As
Rubin (2020, Table 1) summarized, preregistration helps to identify and/or curtail the following issues:
# Poorly planned hypotheses and tests
#
HARKing
HARKing (hypothesizing after the results are known) is an acronym coined by social psychologist Norbert Kerr that refers to the questionable research practice of “presenting a post hoc hypothesis in the introduction of a research report as if it ...
: undisclosed hypothesizing after the results are known
# The suppression of a priori hypotheses that yield null or disconfirming results
# Deviations from planned analyses
# Lack of clarity between confirmatory and exploratory analyses
# Undisclosed
multiple testing
In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values.
The more inferences ...
# Forking paths, in which researchers make decisions about which tests to conduct based on information from their sample
#
''p''-hacking: continuing data analysis until a significant ''p'' value is obtained
# Optional stopping: repeating the same test at different stages of data collection until a significant result is obtained
# Invalid use of
''p'' values, because ''p'' values lose their meaning in exploratory analyses
# Researchers’ biases, including the
confirmation bias
Confirmation bias is the tendency to search for, interpret, favor, and recall information in a way that confirms or supports one's prior beliefs or values. People display this bias when they select information that supports their views, ignoring ...
and
hindsight bias
Hindsight bias, also known as the knew-it-all-along phenomenon or creeping determinism, is the common tendency for people to perceive past events as having been more predictable than they actually were. People often believe that after an event ha ...
# Selective reporting of results: “cherry-picking” specific supportive results and suppressing non-supportive results
# Unclear test severity, preventing the identification of hypotheses that have a low probability of being confirmed when they are false
# Unreported
null findings
#
Publication bias
In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance o ...
: unpublished null findings, resulting in the file draw problem
# Potentially
low replicability, ostensibly due to the use of questionable research practices (e.g., HARKing, ''p''-hacking, optional stopping)
Identifying issues such as these via preregistration helps to improve "the interpretability and credibility of research findings" (
Nosek et al., 2018, p. 2605).
However,
Rubin (2020) argued that only ''some'' of these issues are problematic and only under ''some'' conditions.
He also argued that, when they are problematic, preregistration is not necessary to identify these issues. Instead, they can be identified via (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.
Criticisms
Proponents of preregistration have argued that it is "a method to increase the credibility of published results" (
Nosek & Lakens, 2014), that it "makes your science better by increasing the credibility of your results"
Centre for Open Science, and that it "improves the interpretability and credibility of research findings" (
Nosek et al., 2018, p. 2605).
This argument assumes that non-preregistered ''exploratory'' analyses are less "credible" and/or "interpretable" than preregistered ''confirmatory'' analyses because they may involve "circular reasoning" in which post hoc hypotheses are based on the observed data (
Nosek et al., 2018, p. 2600).
However, critics have argued that preregistration is not necessary to identify circular reasoning during exploratory analyses (
Rubin, 2020). Circular reasoning can be identified by analysing the reasoning per se without needing to know whether that reasoning was preregistered. Critics have also noted that the idea that preregistration improves research credibility may deter researchers from undertaking non-preregistered exploratory analyses (
Coffman & Niederle, 2015; see also
Collins et al., 2021, Study 1).
In response, preregistration advocates have stressed that exploratory analyses are permitted in preregistered studies, and that the results of these analyses retain some value vis-a-vis hypothesis generation rather than hypothesis testing. Preregistration merely makes the distinction between confirmatory and exploratory research clearer (
Nosek et al., 2018;
Nosek & Lakens, 2014;
Wagenmakers et al., 2012).
Hence, although preregistraton is supposed to reduce
researcher degrees of freedom
Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a scientific experiment, and in analyzing its results. The term reflects the fact that researchers can choose betwe ...
during the data analysis stage, it is also supposed to be “a plan, not a prison”
Dehaven, 2017.
However, critics counterargue that, if preregistration is only supposed to be a plan, and not a prison, then researchers should feel free to deviate from that plan and undertake exploratory analyses without fearing accusations of low research credibility due to circular reasoning and inappropriate research practices such as ''p''-hacking and unreported
multiple testing
In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values.
The more inferences ...
that leads to inflated
familywise error rates (e.g.,
Navarro, 2020).
Again, they have pointed out that preregistration is not necessary to address such concerns. For example, concerns about ''p''-hacking and unreported multiple testing can be addressed if researchers engage in other
open science practices, such as (a) open data and research materials and (b) robustness or multiverse analyses
(Rubin, 2020;
Steegen et al., 2016; for several other approaches, se
Srivastava, 2018.
Finally, and more fundamentally, critics have argued that the distinction between confirmatory and exploratory analyses is unclear and/or irrelevant (
Devezer et al., 2020;
Rubin, 2020;
Szollosi & Donkin, 2019),
and that concerns about inflated familywise error rates are unjustified when those error rates refer to abstract, atheoretical studywise hypotheses that are not being tested (Rubin,
2020
2020 was heavily defined by the COVID-19 pandemic, which led to global Social impact of the COVID-19 pandemic, social and Economic impact of the COVID-19 pandemic, economic disruption, mass cancellations and postponements of events, COVID- ...
,
2021
File:2021 collage V2.png, From top left, clockwise: the James Webb Space Telescope was launched in 2021; Protesters in Yangon, Myanmar following the 2021 Myanmar coup d'état, coup d'état; A civil demonstration against the October–November 2021 ...
;
Szollosi et al., 2020).
There are also concerns about the practical implementation of preregistration. Many preregistered protocols leave plenty of room for ''p''-hacking (
Bakker et al., 2020Heirene et al., 2021Ikeda et al., 2019 Singh et al., 2021),
and researchers rarely follow the exact research methods and analyses that they preregister
Abrams et al., 2020 Claesen et al., 2019Heirene et al., 2021 see also
Boghdadly et al., 2018;
Singh et al., 2021Sun et al., 2019.
For example, a survey of 27 preregistered studies found that researchers deviated from their preregistered plans in all cases (
Claesen et al., 2019).
The most frequent deviations were with regards to the planned sample size, exclusion criteria, and statistical model. Hence, what were intended as preregistered confirmatory tests ended up as unplanned exploratory tests. Again, preregistration advocates argue that deviations from preregistered plans are acceptable as long as they are reported transparently and justified. They also point out that even vague preregistrations help to reduce
researcher degrees of freedom
Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a scientific experiment, and in analyzing its results. The term reflects the fact that researchers can choose betwe ...
and make any residual flexibility transparent (
Simmons et al., 2021, p. 180).
However, critics have argued that it is not useful to identify or justify deviations from preregistered plans when those plans do not reflect ''high quality theory and research practice''. As
Rubin (2020) explained, “we should be more interested in the rationale for the ''current'' method and analyses than in the rationale for ''historical'' changes that have led up to the current method and analyses” (pp. 378–379).
Finally, some commentators have argued that, under some circumstances, preregistration may actually harm science by providing a false sense of credibility to research studies and analyses (
Devezer et al., 2020;
McPhetres, 2020;
Pham & Oh, 2020;
Szollosi et al., 2020).
Consistent with this view, there is some evidence that researchers view registered reports as being more credible than standard reports on a range of dimensions
Soderberg et al., 2020 see als
Field et al., 2020for inconclusive evidence),
although it is unclear whether this represents a "false" sense of credibility due to pre-existing positive community attitudes about preregistration or a genuine causal effect of registered reports on quality of research.
See also
*
AllTrials
AllTrials (sometimes called All Trials or AllTrials.net) is a project advocating that clinical research adopt the principles of open research. The project summarizes itself as "All trials registered, all results reported": that is, all clinical tr ...
*
Clinical trial registration
Preregistration is the practice of registering the hypotheses, methods, and/or analyses of a scientific study before it is conducted. This can include analyzing primary data or secondary data. Clinical trial registration is similar, although it may ...
*
Metascience
Metascience (also known as meta-research) is the use of scientific methodology to study science itself. Metascience seeks to increase the quality of scientific research while reducing inefficiency. It is also known as "''research on research''" ...
*
Open science
References
{{reflist
External links
Preregistration resources from the Centre for Open ScienceGuidelines for creating registered reports by the Center for Open ScienceAs Predicted
__FORCETOC__
Peer review
Research
Metascience
Open science
Evidence-based practices