Lord's formulation
Responses
There have been many attempts and interpretations of the paradox, along with its relationship to otherImportance of modeling assumptions
Bock (1975)
Bock responded to the paradox by positing that both statisticians in the scenario are correct once the question being asked is clarified. The first statistician (who compares group means and distributions) is asking "are there differences in average weight gain?", whereas the second is asking "what are the differences in individual weight gain?"Cox and McCullagh (1982)
Cox and McCullagh interpret the problem by constructing a model of what could have happened had the students not dined in the dining hall, where they assume that a student's weight would have stayed constant. They conclude that in fact the first statistician was right when asking about group differences, while the second was right when asking about the effect on an individual.Holland and Rubin (1983)
Holland & Rubin (1983) argue that both statisticians have captured accurate descriptive features of the data: Statistician 1 accurately finds no difference in relative weight changes across the two genders, while Statistician 2 accurately finds a larger average weight gain for boys conditional on a boy and girl have the same starting weight. However, when turning these descriptions into causal statements, they implicitly assert that weight would have otherwise stayed constant (Statistician 1) or that it would have followed the posited linear model (Statistician 2).“In summary, we believe that the following views resolve Lord's Paradox. If both statisticians made only descriptive statements, they would both be correct. Statistician 1 makes their unconditional descriptive statements that the average weight gains for males and females are equal; Statistician 2 makes the conditional (on ''X'') statement that for males and females of equal September weight, the males gain more than the females. In contrast, if the statisticians turned these descriptive statements into causal statements, neither would be correct or incorrect because untestable assumptions determine the correctness of causal statements... Statistician 1 is wrong because he makes a causal statement without specifying the assumption needed to make it true. Statistician 2 is more cautious, since he makes only a descriptive statement. However, unless he too makes further assumptions, his descriptive statement is completely irrelevant to the campus dietician's interest in the effect of the dining hall diet." (pg. 19)Moreover, the underlying assumptions necessary to turn descriptive statements into causal statements are untestable. Unlike descriptive statements (e.g. "the average height in the US is ''X''"), causal statements involve a comparison between what happened and what ''would have'' happened absent an intervention. The latter is unobservable in the real world, a fact that Holland & Rubin term "the fundamental problem of causal inference" (pg. 10). This is explains why researchers often turn to experiments: while we still never observe both counterfactuals for a single subject, experiments let us make statistical claims about these differences in the population under minimal assumptions. Absent an experiment, modelers should carefully describe the model they use to make causal statements and justify those models as strongly as possible.
Pearl (2016)
Pearl (2016) agrees with Lord’s conclusion that the answer cannot be found in the data, but he finds Holland and Rubin’s account to be incomplete. In his views, a complete resolution of the Paradox should provide an answer to Lord’s essential question: "How to allow for preexisting differences between groups?" Moreover, since the answer depends on the causal model assumed, we should explain: (1) Why people find Lord’s story to be "Paradoxical" rather than "In need of more information" and, (2) How to properly utilize causal models to answer Lord’s question, regardless of whether they are testable or not. To this end, Pearl used a simplified version of Lord’s Paradox, proposed by Wainer and Brown,Wainer, H.; Brown, L. (2007) "Three statistical paradoxes in the interpretation of group differences: Illustrated with medical school admission and licensing data" In Rao, C.; Sinharay, S. (Eds.) ''Handbook of Statistics 26: Psychometrics'' Vol. 26 North Holland: Elsevier B.V., pp. 893-918. in which gender differences are not considered. Instead, the quantity of interest is the effect of diet on weight gain, as shown in Figure 2(a). The two ellipses represent two dining halls, each serving a different diet, and each point represents a student's initial and final weights. Note that students who weigh more in the beginning tend to eat in dining hall B, while the ones who weigh less eat in dining hall A. The first statistician claims that switching from Diet A to B would have no effect on weight gain, since the gain ''WF'' – ''WI'' has the same distribution in both ellipses. The second statistician compares the final weights under Diet A to those of Diet B for a group of students with same initial weight ''W0'' and finds that latter is larger than the former in every level of ''W0''. He concludes therefore that the students on Diet B gain more than those on Diet A. As before, the data can’t tell us whom to believe, and a causal model must be assumed to settle the issue. One plausible model is shown in Figure 2(b). In this model, ''WI'' is the only confounder of ''D'' and ''WF'', so controlling for ''WI'' is essential for deconfounding the causal effect needed. Assuming this model, the second statistician would be correct and the first statistician would be wrong. This analysis also unveils why Lord’s story appears paradoxical, and why generations of statisticians have found it perplexing. According to Pearl, the data triggers a clash between two strong intuitions, both are valid in causal thinking, but not in the non-causal thinking invoked by the first statistician. One intuition claims that, to get the needed effect, we must make “proper allowances” for uncontrolled preexisting differences between groups” (i.e. initial weights). The second claims that the overall effect (of Diet on Gain) is just the average of the stratum-specific effects. The two intuitions are valid, but seem to clash when we interpret the first statistician’s finding as zero effect when, in fact, his finding merely entails equality of distributions, and says nothing about ``effects.'' This can also be seen from Figure 2(b), which allows ''D'' to causally affect ''Y'' while, simultaneously, be statistically independent of it (due to path cancelations). This resolution of Lord’s Paradox answers both questions: (1) How to allow for preexisting differences between groups and (2) Why the data appear paradoxical. Pearl's ''do''-calculus further answers question (1) for anyInitial weight as a mediator
Going back to Lord's original problem of comparing boys and girls, Pearl (2016) posits another causal model where sex and initial weight both influence the final weight. Moreover, since sex also influences the initial weight, Initial Weight becomes a mediating variable: sex influences final weight both through a direct effect and an indirect effect (by influencing initial weight, which then influences final weight). Note that none of these variables are confounders, so controls are not strictly necessary in this model. However, the choice of whether to control for initial weight dictates which effect the researcher is measuring: the first statistician does not control and measures a total effect, while the second does control and measures a direct effect."Cases where total and direct effects differ in sign are commonplace. For example, we are not at all surprised when smallpox inoculation carries risks of fatal reaction, yet reduces overall mortality by eradicating smallpox. The direct effect (fatal reaction) in this case is negative for every stratum of the population, yet the total effect (on mortality) is positive for the population as a whole." (pg 4)Tu, Gunnell, and Gilthorpe (2008) use a similar causal framework, but counter that the conceptualization of direct and total effect is not the best framework in many cases because there are many different variables that ''could be'' controlled for, without an experimental basis that these are separate causal paths. From the text:
"Whilst the total effect of birth weight on BP is not affected by the numbers of intermediate body size variables in the model, the estimation of 'direct' effect differs when different intermediate variables are adjusted for. Unless there is experimental evidence to support the notion that there are indeed different paths of direct and indirect effects from birth weight to BP, we are cautious of using such terminology to label the results from multiple regression, as with model 3. In other words, to determine whether the unconditional or conditional relationship reflects the true physiological relationship between birth weight and blood pressure, experiments in which birth weight and current weight can be manipulated are required in order to estimate the impact of birth weight on blood pressure." (pg8)Yu-Kang Tu, David Gunnell, Mark S Gilthorpe. Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon – the reversal paradox. Emerg Themes Epidemiol. 2008; 5: 2.
Relation to other paradoxes
According to Tu, Gunnell, and Gilthorpe, Lord's paradox is the continuous version ofImportance
Broadly, the "fundamental problem of causal inference" and related aggregation conceptsReferences
Notes
{{reflist, group=fn Statistical paradoxes