Fisher's exact test is a
statistical significance test used in the analysis of
contingency tables. Although in practice it is employed when
sample
Sample or samples may refer to:
Base meaning
* Sample (statistics), a subset of a population – complete data set
* Sample (signal), a digital discrete sample of a continuous analog signal
* Sample (material), a specimen or small quantity of s ...
sizes are small, it is valid for all sample sizes. It is named after its inventor,
Ronald Fisher
Sir Ronald Aylmer Fisher (17 February 1890 – 29 July 1962) was a British polymath who was active as a mathematician, statistician, biologist, geneticist, and academic. For his work in statistics, he has been described as "a genius who ...
, and is one of a class of
exact test
In statistics, an exact (significance) test is a test such that if the null hypothesis is true, then all assumptions made during the derivation of the distribution of the test statistic are met. Using an exact test provides a significance test ...
s, so called because the significance of the deviation from a
null hypothesis
In scientific research, the null hypothesis (often denoted ''H''0) is the claim that no difference or relationship exists between two sets of data or variables being analyzed. The null hypothesis is that any experimentally observed difference is d ...
(e.g.,
P-value) can be calculated exactly, rather than relying on an approximation that becomes exact in the limit as the sample size grows to infinity, as with many statistical tests.
Fisher is said to have devised the test following a comment from
Muriel Bristol
Blanche Muriel Bristol (21 April 1888 – 15 March 1950) was a British phycologist who worked at Rothamsted Research (then Rothamsted Experimental Station) in 1919. Her research focused on the mechanisms by which algae acquire nutrients.
Stati ...
, who claimed to be able to detect whether the tea or the milk was added first to her cup. He tested her claim in the "
lady tasting tea
In the design of experiments in statistics, the lady tasting tea is a randomized experiment devised by Ronald Fisher and reported in his book ''The Design of Experiments'' (1935). The experiment is the original exposition of Fisher's notion of ...
" experiment.
Purpose and scope
![Nice Cup of Tea](https://upload.wikimedia.org/wikipedia/commons/3/37/Nice_Cup_of_Tea.jpg)
The test is useful for
categorical data
In statistics, a categorical variable (also called qualitative variable) is a variable that can take on one of a limited, and usually fixed, number of possible values, assigning each individual or other unit of observation to a particular group or ...
that result from classifying objects in two different ways; it is used to examine the significance of the association (contingency) between the two kinds of classification. So in Fisher's original example, one criterion of classification could be whether milk or tea was put in the cup first; the other could be whether Bristol thinks that the milk or tea was put in first. We want to know whether these two classifications are associated—that is, whether Bristol really can tell whether milk or tea was poured in first. Most uses of the Fisher test involve, like this example, a 2 × 2 contingency table (discussed below). The
p-value from the test is computed as if the margins of the table are fixed, i.e. as if, in the tea-tasting example, Bristol knows the number of cups with each treatment (milk or tea first) and will therefore provide guesses with the correct number in each category. As pointed out by Fisher, this leads under a null hypothesis of independence to a
hypergeometric distribution
In probability theory and statistics, the hypergeometric distribution is a discrete probability distribution that describes the probability of k successes (random draws for which the object drawn has a specified feature) in n draws, ''without'' ...
of the numbers in the cells of the table.
With large samples, a
chi-squared test
A chi-squared test (also chi-square or test) is a statistical hypothesis test used in the analysis of contingency tables when the sample sizes are large. In simpler terms, this test is primarily used to examine whether two categorical variables ...
(or better yet, a
G-test
In statistics, ''G''-tests are likelihood ratio test, likelihood-ratio or maximum likelihood statistical significance tests that are increasingly being used in situations where chi-squared tests were previously recommended.
The general formula fo ...
) can be used in this situation. However, the significance value it provides is only an approximation, because the
sampling distribution
In statistics, a sampling distribution or finite-sample distribution is the probability distribution of a given random-sample-based statistic. If an arbitrarily large number of samples, each involving multiple observations (data points), were sep ...
of the test statistic that is calculated is only approximately equal to the theoretical chi-squared distribution. The approximation is inadequate when sample sizes are small, or the data are very unequally distributed among the cells of the table, resulting in the cell counts predicted on the null hypothesis (the "expected values") being low. The usual rule for deciding whether the chi-squared approximation is good enough is that the chi-squared test is not suitable when the expected values in any of the cells of a contingency table are below 5, or below 10 when there is only one
degree of freedom
Degrees of freedom (often abbreviated df or DOF) refers to the number of independent variables or parameters of a thermodynamic system. In various scientific fields, the word "freedom" is used to describe the limits to which physical movement or ...
(this rule is now known to be overly conservative
). In fact, for small, sparse, or unbalanced data, the exact and asymptotic ''p''-values can be quite different and may lead to opposite conclusions concerning the hypothesis of interest.
[Mehta, C. R. 1995. SPSS 6.1 Exact test for Windows. Englewood Cliffs, NJ: Prentice Hall.] In contrast the Fisher exact test is, as its name states, exact as long as the experimental procedure keeps the row and column totals fixed, and it can therefore be used regardless of the sample characteristics. It becomes difficult to calculate with large samples or well-balanced tables, but fortunately these are exactly the conditions where the chi-squared test is appropriate.
For hand calculations, the test is feasible only in the case of a 2 × 2 contingency table. However the principle of the test can be extended to the general case of an ''m'' × ''n'' table, and some
statistical packages provide a calculation (sometimes using a
Monte Carlo method
Monte Carlo methods, or Monte Carlo experiments, are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. The underlying concept is to use randomness to solve problems that might be determi ...
to obtain an approximation) for the more general case.
The test can also be used to quantify the ''overlap'' between two sets.
For example, in enrichment analyses in statistical genetics one set of genes may be annotated for a given phenotype and the user may be interested in testing the overlap of their own set with those.
In this case a 2 × 2 contingency table may be generated and Fisher's exact test applied through identifying
# Genes that are provided in both lists
# Genes that are provided in the first list and not the second
# Genes that are provided in the second list and not the first
# Genes that are not provided in either list
The test assumes genes in either list are taken from a broader set of genes (e.g. all remaining genes).
A p-value may then be calculated, summarizing the significance of the overlap between the two lists.
Example
For example, a sample of teenagers might be divided into male and female on one hand and those who are and are not currently studying for a statistics exam on the other. For example, we hypothesize that the proportion of studying students is higher among the women than among the men, and we want to test whether any difference in proportions that we observe is significant. The data might look like this:
The question we ask about these data is: Knowing that 10 of these 24 teenagers are studying and that 12 of the 24 are female, and assuming the null hypothesis that men and women are equally likely to study, what is the probability that these 10 teenagers who are studying would be so unevenly distributed between the women and the men? If we were to choose 10 of the teenagers at random, what is the probability that 9 or more of them would be among the 12 women and only 1 or fewer from among the 12 men?
Before we proceed with the Fisher test, we first introduce some notations. We represent the cells by the letters ''a, b, c'' and ''d'', call the totals across rows and columns ''marginal totals'', and represent the grand total by ''n''. So the table now looks like this:
Fisher showed that conditional on the margins of the table, ''a'' is distributed as a
hypergeometric distribution
In probability theory and statistics, the hypergeometric distribution is a discrete probability distribution that describes the probability of k successes (random draws for which the object drawn has a specified feature) in n draws, ''without'' ...
with ''a+c'' draws from a population with ''a+b'' successes and ''c+d'' failures. The probability of obtaining such set of values is given by:
where
is the
binomial coefficient and the symbol ! indicates the
factorial operator.
This can be seen as follows. If the marginal totals (i.e.
,
,
, and
) are known, only a single degree of freedom is left: the value e.g. of
suffices to deduce the other values.
Now,
is the probability that
elements are positive in a random selection (without replacement) of
elements from a larger set containing
elements in total out of which
are positive, which is precisely the definition of the hypergeometric distribution.
With the data above (using the first of the equivalent forms), this gives:
The formula above gives the exact hypergeometric probability of observing this particular arrangement of the data, assuming the given marginal totals, on the
null hypothesis
In scientific research, the null hypothesis (often denoted ''H''0) is the claim that no difference or relationship exists between two sets of data or variables being analyzed. The null hypothesis is that any experimentally observed difference is d ...
that men and women are equally likely to be studiers. To put it another way, if we assume that the probability that a man is a studier is
, the probability that a woman is a studier is also
, and we assume that both men and women enter our sample independently of whether or not they are studiers, then this hypergeometric formula gives the conditional probability of observing the values ''a, b, c, d'' in the four cells, conditionally on the observed marginals (i.e., assuming the row and column totals shown in the margins of the table are given). This remains true even if men enter our sample with different probabilities than women. The requirement is merely that the two classification characteristics—gender, and studier (or not)—are not associated.
For example, suppose we knew probabilities
with
such that (male studier, male non-studier, female studier, female non-studier) had respective probabilities
for each individual encountered under our sampling procedure. Then still, were we to calculate the distribution of cell entries conditional given marginals, we would obtain the above formula in which neither
nor
occurs. Thus, we can calculate the exact probability of any arrangement of the 24 teenagers into the four cells of the table, but Fisher showed that to generate a significance level, we need consider only the cases where the marginal totals are the same as in the observed table, and among those, only the cases where the arrangement is as extreme as the observed arrangement, or more so. (
Barnard's test
In statistics, Barnard’s test is an exact test used in the analysis of contingency tables with one margin fixed. Barnard’s tests are really a class of hypothesis tests, also known as unconditional exact tests for two independent binomials. ...
relaxes this constraint on one set of the marginal totals.) In the example, there are 11 such cases. Of these only one is more extreme in the same direction as our data; it looks like this:
For this table (with extremely unequal studying proportions) the probability is
.
In order to calculate the significance of the observed data, i.e. the total probability of observing data as extreme or more extreme if the
null hypothesis
In scientific research, the null hypothesis (often denoted ''H''0) is the claim that no difference or relationship exists between two sets of data or variables being analyzed. The null hypothesis is that any experimentally observed difference is d ...
is true, we have to calculate the values of ''p'' for both these tables, and add them together. This gives a
one-tailed test
In statistical significance testing, a one-tailed test and a two-tailed test are alternative ways of computing the statistical significance of a parameter inferred from a data set, in terms of a test statistic. A two-tailed test is appropriate if ...
, with ''p'' approximately 0.001346076 + 0.000033652 = 0.001379728. For example, in the
R statistical computing environment, this value can be obtained as
fisher.test(rbind(c(1,9),c(11,3)), alternative="less")$p.value
, or in python, using
scipy.stats.fisher_exact(table= 1,9 1,3, alternative="less")
(where one receives both the prior odds ratio and the p-value). This value can be interpreted as the sum of evidence provided by the observed data—or any more extreme table—for the
null hypothesis
In scientific research, the null hypothesis (often denoted ''H''0) is the claim that no difference or relationship exists between two sets of data or variables being analyzed. The null hypothesis is that any experimentally observed difference is d ...
(that there is no difference in the proportions of studiers between men and women). The smaller the value of ''p'', the greater the evidence for rejecting the null hypothesis; so here the evidence is strong that men and women are not equally likely to be studiers.
For a
two-tailed test
In statistical significance testing, a one-tailed test and a two-tailed test are alternative ways of computing the statistical significance of a parameter inferred from a data set, in terms of a test statistic. A two-tailed test is appropriate i ...
we must also consider tables that are equally extreme, but in the opposite direction. Unfortunately, classification of the tables according to whether or not they are 'as extreme' is problematic. An approach used by the
fisher.test
function in
R is to compute the p-value by summing the probabilities for all tables with probabilities less than or equal to that of the observed table. In the example here, the 2-sided p-value is twice the 1-sided value—but in general these can differ substantially for tables with small counts, unlike the case with test statistics that have a symmetric sampling distribution.
As noted above, most modern
statistical packages will calculate the significance of Fisher tests, in some cases even where the chi-squared approximation would also be acceptable. The actual computations as performed by statistical software packages will as a rule differ from those described above, because numerical difficulties may result from the large values taken by the factorials. A simple, somewhat better computational approach relies on a
gamma function
In mathematics, the gamma function (represented by , the capital letter gamma from the Greek alphabet) is one commonly used extension of the factorial function to complex numbers. The gamma function is defined for all complex numbers except ...
or log-gamma function, but methods for accurate computation of hypergeometric and binomial probabilities remains an active research area.
Controversies
Despite the fact that Fisher's test gives exact p-values, some authors have argued that it is conservative, i.e. that its actual rejection rate is below the nominal significance level.
The apparent contradiction stems from the combination of a discrete statistic with fixed significance levels.
To be more precise, consider the following proposal for a significance test at the 5%-level: reject the null hypothesis for each table to which Fisher's test assigns a p-value equal to or smaller than 5%. Because the set of all tables is discrete, there may not be a table for which equality is achieved. If
is the largest p-value smaller than 5% which can actually occur for some table, then the proposed test effectively tests at the
-level. For small sample sizes,
might be significantly lower than 5%.
While this effect occurs for any discrete statistic (not just in contingency tables, or for Fisher's test), it has been argued that the problem is compounded by the fact that Fisher's test conditions on the marginals. To avoid the problem, many authors discourage the use of fixed significance levels when dealing with discrete problems.
The decision to condition on the margins of the table is also controversial.
[
][
;
] The p-values derived from Fisher's test come from the distribution that conditions on the margin totals. In this sense, the test is exact only for the conditional distribution and not the original table where the margin totals may change from experiment to experiment. It is possible to obtain an exact p-value for the 2×2 table when the margins are not held fixed.
Barnard's test
In statistics, Barnard’s test is an exact test used in the analysis of contingency tables with one margin fixed. Barnard’s tests are really a class of hypothesis tests, also known as unconditional exact tests for two independent binomials. ...
, for example, allows for random margins. However, some authors
(including, later, Barnard himself)
have criticized Barnard's test based on this property. They argue that the marginal success total is an (almost
)
ancillary statistic An ancillary statistic is a measure of a sample whose distribution (or whose pmf or pdf) does not depend on the parameters of the model. An ancillary statistic is a pivotal quantity that is also a statistic. Ancillary statistics can be used to c ...
, containing (almost) no information about the tested property.
The act of conditioning on the marginal success rate from a 2×2 table can be shown to ignore some information in the data about the unknown odds ratio.
[
] The argument that the marginal totals are (almost) ancillary implies that the appropriate likelihood function for making inferences about this odds ratio should be conditioned on the marginal success rate.
Whether this lost information is important for inferential purposes is the essence of the controversy.
Alternatives
An alternative exact test,
Barnard's exact test
In statistics, Barnard’s test is an exact test used in the analysis of contingency tables with one margin fixed. Barnard’s tests are really a class of hypothesis tests, also known as unconditional exact tests for two independent binomials. T ...
, has been developed and proponents of it suggest that this method is more powerful, particularly in 2×2 tables.
Furthermore,
Boschloo's test
Boschloo's test is a statistical hypothesis test for analysing 2x2 contingency tables. It examines the association of two Bernoulli distributed random variables and is a uniformly more powerful alternative to Fisher's exact test. It was propo ...
is an exact test that is uniformly more powerful than Fisher's exact test by construction.
Another alternative is to use
maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed stat ...
estimates to calculate a
p-value from the exact
binomial
Binomial may refer to:
In mathematics
*Binomial (polynomial), a polynomial with two terms
* Binomial coefficient, numbers appearing in the expansions of powers of binomials
*Binomial QMF, a perfect-reconstruction orthogonal wavelet decomposition
...
or
multinomial distributions and reject or fail to reject based on the
p-value.
For stratified categorical data the
Cochran–Mantel–Haenszel test must be used instead of Fisher's test.
Choi et al.
propose a p-value derived from the likelihood ratio test based on the conditional distribution of the
odds ratio
An odds ratio (OR) is a statistic that quantifies the strength of the association between two events, A and B. The odds ratio is defined as the ratio of the odds of A in the presence of B and the odds of A in the absence of B, or equivalently (due ...
given the marginal success rate. This p-value is inferentially consistent with classical tests of normally distributed data as well as with likelihood ratios and support intervals based on this conditional likelihood function. It is also readily computable.
[
See also]
Likelihood Ratio Statistics for 2 x 2 Tables
(Online calculator).
See also
*
Bernoulli trial
In the theory of probability and statistics, a Bernoulli trial (or binomial trial) is a random experiment with exactly two possible outcomes, "success" and "failure", in which the probability of success is the same every time the experiment is ...
*
Boschloo's test
Boschloo's test is a statistical hypothesis test for analysing 2x2 contingency tables. It examines the association of two Bernoulli distributed random variables and is a uniformly more powerful alternative to Fisher's exact test. It was propo ...
References
External links
Calculate Fisher's Exact Test OnlineLikelihood Ratio Statistics for 2X2 Tables
{{DEFAULTSORT:Fisher's Exact Test
Statistical tests for contingency tables
Nonparametric statistics