Researcher degrees of freedom is a concept referring to the inherent flexibility involved in the process of designing and conducting a
scientific experiment
An experiment is a procedure carried out to support or refute a hypothesis, or determine the efficacy or likelihood of something previously untried. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a ...
, and in analyzing its results. The term reflects the fact that researchers can choose between multiple ways of collecting and analyzing data, and these decisions can be made either arbitrarily or because they, unlike other possible choices, produce a positive and
statistically significant result.
As such, researcher degrees of freedom are often related to
data dredging and other questionable research practices. Their widespread use represents an inherent methodological limitation in
scientific research
The scientific method is an empirical method for acquiring knowledge that has characterized the development of science since at least the 17th century (with notable practitioners in previous centuries; see the article history of scientific m ...
, and contributes to an inflated rate of
false-positive
A false positive is an error in binary classification in which a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error, where the test result ...
findings.
They can also lead to overestimated
effect size
In statistics, an effect size is a value measuring the strength of the relationship between two variables in a population, or a sample-based estimate of that quantity. It can refer to the value of a statistic calculated from a sample of data, the ...
s.
Though the concept of researcher degrees of freedom has mainly been discussed in the context of
psychology
Psychology is the scientific study of mind and behavior. Psychology includes the study of conscious and unconscious phenomena, including feelings and thoughts. It is an academic discipline of immense scope, crossing the boundaries between ...
, it can affect any scientific discipline.
Like
publication bias, the existence of researcher degrees of freedom has the potential to lead to an inflated degree of
funnel plot
A funnel plot is a graph designed to check for the existence of publication bias; funnel plots are commonly used in systematic reviews and meta-analyses. In the absence of publication bias, it assumes that studies with high precision will be plotte ...
asymmetry. Furthermore, studies with smaller
sample size
Sample size determination is the act of choosing the number of observations or replicates to include in a statistical sample. The sample size is an important feature of any empirical study in which the goal is to make inferences about a populatio ...
s are more susceptible to the biasing influence of researcher degrees of freedom.
Examples
Steegen et al. (2016) showed how, starting from a single raw data set, applying different reasonable data processing decisions can give rise to a multitude of processed data sets (called the data multiverse), often leading to different statistical results.
Wicherts et al. (2016) provided a list of 34 degrees of freedom (DFs) researchers have when conducting psychological research. The DFs listed span every stage of the research process, from formulating a
hypothesis
A hypothesis (plural hypotheses) is a proposed explanation for a phenomenon. For a hypothesis to be a scientific hypothesis, the scientific method requires that one can test it. Scientists generally base scientific hypotheses on previous obse ...
to the reporting of results. They include conducting exploratory, hypothesis-free research, which the authors note "...pervades many of the researcher DFs that we describe below in the later phases of the study." Other DFs listed in this paper include the creation of multiple manipulated
independent variables and the measurement of additional variables that may be selected for analysis later on.
See also
*
Overfitting
References
Bias
Academic publishing
{{Statistics-stub