In
statistics
Statistics (from German language, German: ''wikt:Statistik#German, Statistik'', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of ...
, the Holm–Bonferroni method, also called the Holm method or Bonferroni–Holm method, is used to counteract the problem of
multiple comparisons
In statistics, the multiple comparisons, multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values.
The more inferences ...
. It is intended to control the
family-wise error rate
In statistics, family-wise error rate (FWER) is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests.
Familywise and Experimentwise Error Rates
Tukey (1953) developed the concept of a ...
(FWER) and offers a simple test
uniformly more powerful than the
Bonferroni correction
In statistics, the Bonferroni correction is a method to counteract the multiple comparisons problem.
Background
The method is named for its use of the Bonferroni inequalities.
An extension of the method to confidence intervals was proposed by Oliv ...
. It is named after
Sture Holm, who codified the method, and
Carlo Emilio Bonferroni
Carlo Emilio Bonferroni (28 January 1892 – 18 August 1960) was an Italian mathematician who worked on probability theory.
Biography
Bonferroni studied piano and conducting in Turin Conservatory and at University of Turin under Giuseppe Peano ...
.
Motivation
When considering several hypotheses, the problem of
multiplicity
Multiplicity may refer to: In science and the humanities
* Multiplicity (mathematics), the number of times an element is repeated in a multiset
* Multiplicity (philosophy), a philosophical concept
* Multiplicity (psychology), having or using multi ...
arises: the more hypotheses are checked, the higher the probability of obtaining
Type I error
In statistical hypothesis testing, a type I error is the mistaken rejection of an actually true null hypothesis (also known as a "false positive" finding or conclusion; example: "an innocent person is convicted"), while a type II error is the fa ...
s (
false positive
A false positive is an error in binary classification in which a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error, where the test result ...
s). The Holm–Bonferroni method is one of many approaches for controlling the FWER, i.e., the probability that one or more Type I errors will occur, by adjusting the rejection criteria for each of the individual hypotheses.
Formulation
The method is as follows:
* Suppose you have
p-values, sorted into order lowest-to-highest
, and their corresponding hypotheses
(null hypotheses). You want the FWER to be no higher than a certain pre-specified
significance level
In statistical hypothesis testing, a result has statistical significance when it is very unlikely to have occurred given the null hypothesis (simply by chance alone). More precisely, a study's defined significance level, denoted by \alpha, is the ...
.
* Is
? If so, reject
and continue to the next step, otherwise EXIT.
* Is
? If so, reject
also, and continue to the next step, otherwise EXIT.
* And so on: for each P value, test whether
. If so, reject
and continue to examine the larger P values, otherwise EXIT.
This method ensures that the FWER is at most
, in the strong sense.
Rationale
The simple
Bonferroni correction
In statistics, the Bonferroni correction is a method to counteract the multiple comparisons problem.
Background
The method is named for its use of the Bonferroni inequalities.
An extension of the method to confidence intervals was proposed by Oliv ...
rejects only null hypotheses with ''p''-value less than
, in order to ensure that the FWER, i.e., the risk of rejecting one or more true null hypotheses (i.e., of committing one or more type I errors) is at most
. The cost of this protection against type I errors is an increased risk of failing to reject one or more false null hypotheses (i.e., of committing one or more type II errors).
The Holm–Bonferroni method also controls the FWER at
, but with a lower increase of type II error risk than the classical Bonferroni method. The Holm–Bonferroni method sorts the ''p''-values from lowest to highest and compares them to nominal alpha levels of
to
(respectively), namely the values
.
* The index
identifies the first ''p''-value that is ''not'' low enough to validate rejection. Therefore, the null hypotheses
are rejected, while the null hypotheses
are not rejected.
* If
then no ''p''-values were low enough for rejection, therefore no null hypotheses are rejected.
* If no such index
could be found then all ''p''-values were low enough for rejection, therefore all null hypotheses are rejected (none are accepted).
Proof
Holm–Bonferroni controls the FWER as follows. Let
be a family of hypotheses, and
be the sorted p-values. Let
be the set of indices corresponding to the (unknown) true null hypotheses, having
members.
Let us assume that we wrongly reject a true hypothesis. We have to prove that the probability of this event is at most
. Let
be such that
is the first rejected true hypothesis, in the ordering used during the Bonferroni–Holm test. Then
are all rejected false hypotheses. It then holds that
and
(1). Since
is rejected, it must be
by definition of the testing procedure. Using (1), the right hand side of this inequality is at most
. Thus, if we wrongly reject a true hypothesis, there has to be a true hypothesis with P-value at most
.
So let us define the random variable
. Whatever the (unknown) set of true hypotheses
is, we have
(by the
Bonferroni inequalities
In probability theory, Boole's inequality, also known as the union bound, says that for any finite or countable set of events, the probability that at least one of the events happens is no greater than the sum of the probabilities of the individu ...
). Therefore, the probability to reject a true hypothesis is at most
.
Alternative proof
The Holm–Bonferroni method can be viewed as a
closed testing procedure
In statistics, the closed testing procedure is a general method for performing more than one hypothesis test simultaneously.
The closed testing principle
Suppose there are ''k'' hypotheses ''H''1,..., ''H'k'' to be tested and the overall type I ...
,
with the Bonferroni correction applied locally on each of the intersections of null hypotheses.
The closure principle states that a hypothesis
in a family of hypotheses
is rejected – while controlling the FWER at level
– if and only if all the sub-families of the intersections with
are rejected at level
.
The Holm-Bonferroni method is a ''shortcut procedure'', since it makes
or less comparisons, while the number of all intersections of null hypotheses to be tested is of order
.
It controls the FWER in the strong sense.
In the Holm–Bonferroni procedure, we first test
. If it is not rejected then the intersection of all null hypotheses
is not rejected too, such that there exists at least one intersection hypothesis for each of elementary hypotheses
that is not rejected, thus we reject none of the elementary hypotheses.
If
is rejected at level
then all the intersection sub-families that contain it are rejected too, thus
is rejected.
This is because
is the smallest in each one of the intersection sub-families and the size of the sub-families is at most
, such that the Bonferroni threshold larger than
.
The same rationale applies for
. However, since
already rejected, it sufficient to reject all the intersection sub-families of
without
. Once
holds all the intersections that contains
are rejected.
The same applies for each
.
Example
Consider four null hypotheses
with unadjusted p-values
,
,
and
, to be tested at significance level
. Since the procedure is step-down, we first test
, which has the smallest p-value
. The p-value is compared to
, the null hypothesis is rejected and we continue to the next one. Since
we reject
as well and continue. The next hypothesis
is not rejected since
. We stop testing and conclude that
and
are rejected and
and
are not rejected while controlling the family-wise error rate at level
. Note that even though
applies,
is not rejected. This is because the testing procedure stops once a failure to reject occurs.
Extensions
Holm–Šidák method
When the hypothesis tests are not negatively dependent, it is possible to replace
with:
:
resulting in a slightly more powerful test.
Weighted version
Let
be the ordered unadjusted p-values. Let
,
correspond to
. Reject
as long as
:
Adjusted ''p''-values
The adjusted
''p''-values for Holm–Bonferroni method are:
:
In the earlier example, the adjusted ''p''-values are
,
,
and
. Only hypotheses
and
are rejected at level
.
Similar adjusted ''p''-values for Holm-Šidák method can be defined recursively as
, where
. Due to the inequality
for
, the Holm-Šidák method will be more powerful than Holm-Bonferroni method.
The weighted adjusted ''p''-values are:
:
A hypothesis is rejected at level α if and only if its adjusted ''p''-value is less than α. In the earlier example using equal weights, the adjusted ''p''-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.
Alternatives and usage
The Holm–Bonferroni method is "uniformly" more powerful than the classic
Bonferroni correction
In statistics, the Bonferroni correction is a method to counteract the multiple comparisons problem.
Background
The method is named for its use of the Bonferroni inequalities.
An extension of the method to confidence intervals was proposed by Oliv ...
, meaning that it is always at least as powerful.
There are other methods for controlling the FWER that are more powerful than Holm–Bonferroni. For instance, in the
Hochberg procedure, rejection of
is made after finding the ''maximal'' index
such that
. Thus, The Hochberg procedure is uniformly more powerful than the Holm procedure. However, the Hochberg procedure requires the hypotheses to be
independent
Independent or Independents may refer to:
Arts, entertainment, and media Artist groups
* Independents (artist group), a group of modernist painters based in the New Hope, Pennsylvania, area of the United States during the early 1930s
* Independ ...
or under certain forms of positive dependence, whereas Holm–Bonferroni can be applied without such assumptions. A similar step-up procedure is the Hommel procedure, which is uniformly more powerful than the Hochberg procedure.
Naming
Carlo Emilio Bonferroni did not take part in inventing the method described here. Holm originally called the method the "sequentially rejective Bonferroni test", and it became known as Holm–Bonferroni only after some time. Holm's motives for naming his method after Bonferroni are explained in the original paper:
''"The use of the Boole inequality within multiple inference theory is usually called the Bonferroni technique, and for this reason we will call our test the sequentially rejective Bonferroni test."''
References
{{DEFAULTSORT:Holm-Bonferroni Method
Statistical hypothesis testing
Statistical tests
Multiple comparisons