Misunderstandings of p-values
   HOME

TheInfoList



OR:

Misuse of ''p''-values is common in
scientific research The scientific method is an empirical method for acquiring knowledge that has characterized the development of science since at least the 17th century (with notable practitioners in previous centuries; see the article history of scientific m ...
and
scientific education Science education is the teaching and learning of science to school children, college students, or adults within the general public. The field of science education includes work in science content, science process (the scientific method), some ...
. ''p''-values are often used or interpreted incorrectly; the
American Statistical Association The American Statistical Association (ASA) is the main professional organization for statisticians and related professionals in the United States. It was founded in Boston, Massachusetts on November 27, 1839, and is the second oldest continuousl ...
states that ''p''-values can indicate how incompatible the data are with a specified statistical model. From a Neyman–Pearson hypothesis testing approach to statistical inferences, the data obtained by comparing the ''p''-value to a
significance level In statistical hypothesis testing, a result has statistical significance when it is very unlikely to have occurred given the null hypothesis (simply by chance alone). More precisely, a study's defined significance level, denoted by \alpha, is the ...
will yield one of two results: either the
null hypothesis In scientific research, the null hypothesis (often denoted ''H''0) is the claim that no difference or relationship exists between two sets of data or variables being analyzed. The null hypothesis is that any experimentally observed difference is d ...
is rejected (which however does not prove that the null hypothesis is ''false''), or the null hypothesis ''cannot'' be rejected at that significance level (which however does not prove that the null hypothesis is ''true''). From a Fisherian statistical testing approach to statistical inferences, a low ''p''-value means ''either'' that the null hypothesis is true and a highly improbable event has occurred ''or'' that the null hypothesis is false.


Clarifications about ''p''-values

The following list clarifies some issues that are commonly misunderstood regarding ''p''-values: #The ''p''-value is ''not'' the probability that the null hypothesis is true, or the probability that the alternative hypothesis is false. A ''p''-value can indicate the degree of compatibility between a dataset and a particular hypothetical explanation (such as a null hypothesis). Specifically, the ''p''-value can be taken as the prior probability of obtaining an effect that is at least as extreme as the observed effect, given that the null hypothesis is true. This should not be confused with the posterior probability that the null hypothesis is true given the observed effect (see
prosecutor's fallacy The prosecutor's fallacy is a fallacy of statistical reasoning involving a test for an occurrence, such as a DNA match. A positive result in the test may paradoxically be more likely to be an erroneous result than an actual occurrence, even i ...
). In fact,
frequentist statistics Frequentist inference is a type of statistical inference based in frequentist probability, which treats “probability” in equivalent terms to “frequency” and draws conclusions from sample-data by means of emphasizing the frequency or pro ...
does not attach probabilities to hypotheses. #The ''p''-value is ''not'' the probability that the observed effects were produced by random chance alone. The ''p''-value is computed under the assumption that a certain model, usually the null hypothesis, is true. This means that the ''p''-value is a statement about the relation of the data to that hypothesis. #The 0.05 significance level is merely a convention. The 0.05 significance level (alpha level) is often used as the boundary between a statistically significant and a statistically non-significant ''p''-value. However, this does not imply that there is generally a scientific reason to consider results on opposite sides of any threshold as qualitatively different. #The ''p''-value does not indicate the size or importance of the observed effect. A small ''p''-value can be observed for an effect that is not meaningful or important. In fact, the larger the sample size, the smaller the minimum effect needed to produce a statistically significant ''p''-value (see
effect size In statistics, an effect size is a value measuring the strength of the relationship between two variables in a population, or a sample-based estimate of that quantity. It can refer to the value of a statistic calculated from a sample of data, the ...
). Visualizing effect sizes is a critical component of a data-analysis method called
estimation statistics Estimation statistics, or simply estimation, is a data analysis framework that uses a combination of effect sizes, confidence intervals, precision planning, and meta-analysis to plan experiments, analyze data and interpret results. It complement ...
.


Representing probabilities of hypotheses

A frequentist approach rejects the validity of representing probabilities of hypotheses: hypotheses are true or false, not something that can be represented with a probability.
Bayesian statistics Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
actively models the likelihood of hypotheses. The ''p''-value does not in itself allow reasoning about the probabilities of hypotheses, which requires multiple hypotheses or a range of hypotheses, with a
prior distribution In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken int ...
of likelihoods between them, in which case Bayesian statistics could be used. There, one uses a
likelihood function The likelihood function (often simply called the likelihood) represents the probability of random variable realizations conditional on particular values of the statistical parameters. Thus, when evaluated on a given sample, the likelihood funct ...
for all possible values of the prior instead of the ''p''-value for a single null hypothesis. The ''p''-value describes a property of data when compared to a specific null hypothesis; it is not a property of the hypothesis itself. For the same reason, ''p''-values do not give the probability that the data were produced by random chance alone.


Multiple comparisons problem

The multiple comparisons problem occurs when one considers a set of
statistical inference Statistical inference is the process of using data analysis to infer properties of an underlying probability distribution, distribution of probability.Upton, G., Cook, I. (2008) ''Oxford Dictionary of Statistics'', OUP. . Inferential statistical ...
s simultaneously or infers a subset of parameters selected based on the observed values. It is also known as the
look-elsewhere effect The look-elsewhere effect is a phenomenon in the statistical analysis of scientific experiments where an apparently statistically significant observation may have actually arisen by chance because of the sheer size of the parameter space to be sear ...
. Errors in inference, including
confidence interval In frequentist statistics, a confidence interval (CI) is a range of estimates for an unknown parameter. A confidence interval is computed at a designated ''confidence level''; the 95% confidence level is most common, but other levels, such as 9 ...
s that fail to include their corresponding population parameters or
hypothesis test A statistical hypothesis test is a method of statistical inference used to decide whether the data at hand sufficiently support a particular hypothesis. Hypothesis testing allows us to make probabilistic statements about population parameters. ...
s that incorrectly reject the
null hypothesis In scientific research, the null hypothesis (often denoted ''H''0) is the claim that no difference or relationship exists between two sets of data or variables being analyzed. The null hypothesis is that any experimentally observed difference is d ...
, are more likely to occur when one considers the set as a whole. Several statistical techniques have been developed to prevent this from happening, allowing significance levels for single and multiple comparisons to be directly compared. These techniques generally require a higher significance threshold for individual comparisons, so as to compensate for the number of inferences being made. The
webcomic Webcomics (also known as online comics or Internet comics) are comics published on a website or mobile app. While many are published exclusively on the web, others are also published in magazines, newspapers, or comic books. Webcomics can be co ...
''
xkcd ''xkcd'', sometimes styled ''XKCD'', is a webcomic created in 2005 by American author Randall Munroe. The comic's tagline describes it as "a webcomic of romance, sarcasm, math, and language". Munroe states on the comic's website that the name ...
'' satirized misunderstandings of ''p''-values by portraying scientists investigating the claim that eating
jellybeans Jelly beans are small bean shaped sugar candies with soft candy shells and thick gel interiors (see gelatin and jelly). The confection is primarily made of sugar and sold in a wide variety of colors and flavors. History It has been claimed ...
caused
acne Acne, also known as ''acne vulgaris'', is a long-term Cutaneous condition, skin condition that occurs when Keratinocyte, dead skin cells and Sebum, oil from the skin clog hair follicles. Typical features of the condition include comedo, black ...
. After failing to find a significant (''p'' < 0.05) correlation between eating jellybeans and acne, the scientists investigate 20 different colors of jellybeans individually, without adjusting for multiple comparisons. They find one color (green) nominally associated with acne (''p'' < 0.05). The results are then reported by a newspaper as indicating that green jellybeans are linked to acne at a 95% confidence level—as if green were the only color tested. In fact, if 20 independent tests are conducted at the 0.05 significance level and all null hypotheses are true, there is a 64.2% chance of obtaining at least one false positive and the
expected number In probability theory, the expected value (also called expectation, expectancy, mathematical expectation, mean, average, or first moment) is a generalization of the weighted average. Informally, the expected value is the arithmetic mean of a ...
of false positives is 1 (i.e. 0.05 × 20). In general, the
family-wise error rate In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests. Familywise and Experimentwise Error Rates Tukey (1953) developed the concept of a ...
(FWER)—the probability of obtaining at least one false positive—increases with the number of tests performed. The FWER when all null hypotheses are true for ''m'' independent tests, each conducted at significance level α, is: :\text=1 - (1-\alpha)^m


See also

*
Estimation statistics Estimation statistics, or simply estimation, is a data analysis framework that uses a combination of effect sizes, confidence intervals, precision planning, and meta-analysis to plan experiments, analyze data and interpret results. It complement ...
*
Replication crisis The replication crisis (also called the replicability crisis and the reproducibility crisis) is an ongoing methodological crisis in which the results of many scientific studies are difficult or impossible to reproduce. Because the reproducibili ...
*
Metascience Metascience (also known as meta-research) is the use of scientific methodology to study science itself. Metascience seeks to increase the quality of scientific research while reducing inefficiency. It is also known as "''research on research''" ...
*
Misuse of statistics Statistics, when used in a misleading fashion, can trick the casual observer into believing something other than what the data shows. That is, a misuse of statistics occurs when a statistical argument asserts a falsehood. In some cases, the misus ...
*
Statcheck Statcheck is an R package designed to detect statistical errors in peer-reviewed psychology articles by searching papers for statistical results, redoing the calculations described in each paper, and comparing the two values to see if they match. I ...


References


Further reading

* * * * * * {{refend Statistical hypothesis testing Probability fallacies