Study heterogeneity

In statistics, (between-) study heterogeneity is a phenomenon that commonly occurs when attempting to undertake a meta-analysis. In a simplistic scenario, studies whose results are to be combined in the meta-analysis would all be undertaken in the same way and to the same experimental protocols. Differences between outcomes would only be due to measurement error (and studies would hence be ''homogeneous''). Study heterogeneity denotes the variability in outcomes that goes beyond what would be expected (or could be explained) due to measurement error alone.

Introduction

Meta-analysis is a method used to combine the results of different trials in order to obtain a quantitative synthesis. The size of individual clinical trials is often too small to detect treatment effects reliably. Meta-analysis increases the power of statistical analyses by pooling the results of all available trials.

As one tries to use meta-analysis to estimate a combined effect from a group of similar studies, the effects found in the individual studies need to be similar enough that one can be confident that a combined estimate will be a meaningful description of the set of studies. However, the individual estimates of treatment effect will vary by chance; some variation is expected due to observational error. Any excess variation (whether it is apparent or detectable or not) is called (statistical) heterogeneity.
The presence of some heterogeneity is not unusual, e.g., analogous effects are also commonly encountered even ''within'' studies, in multicenter trials (between-''center'' heterogeneity).

Reasons for the additional variability are usually differences in the studies themselves, the investigated populations, treatment schedules, endpoint definitions, or other circumstances (clinical diversity). Different types of effect measures (e.g., odds ratio vs. relative risk) may also be more or less susceptible to heterogeneity.

Modeling

In case the origin of heterogeneity can be identified and may be attributed to certain study features, the analysis may be stratified (by considering subgroups of studies, which would then hopefully be more homogeneous), or by extending the analysis to a meta-regression, accounting for (continuous or categorical) moderator variables. Unfortunately, literature-based meta-analysis may often not allow for gathering data on all (potentially) relevant moderators.

In addition, heterogeneity is usually accommodated by using a random effects model, in which the heterogeneity then constitutes a variance component. The model represents the lack of knowledge about why treatment effects may differ by treating the (potential) differences as unknowns. The centre of this symmetric distribution describes the average of the effects, while its width describes the degree of heterogeneity. The obvious and conventional choice of distribution is a normal distribution. It is difficult to establish the validity of any distributional assumption, and this is a common criticism of random effects meta-analyses. However, variations of the exact distributional form may not make much of a difference, and simulations have shown that methods are relatively robust even under extreme distributional assumptions, both in estimating heterogeneity, and calculating an overall effect size.

Inclusion of a random effect to the model has the effect of making the inferences (in a sense) more conservative or cautious, as a (non-zero) heterogeneity will lead to greater uncertainty (and avoid overconfidence) in the estimation of overall effects. In the special case of a zero heterogeneity variance, the random-effects model again reduces to the special case of the common-effect model.

Common meta-analysis models, however, should of course not be applied blindly or naively to collected sets of estimates. In case the results to be amalgamated differ substantially (in their contexts or in their estimated effects), a derived meta-analytic average may eventually not correspond to a reasonable estimand.
When individual studies exhibit conflicting results, there likely are some reasons why the results differ; for instance, two subpopulations may experience different pharmacokinetic pathways. In such a scenario, it would be important to both know ''and'' consider relevant covariables in an analysis.

Testing

Statistical testing for a non-zero heterogeneity variance is often done based on '' Cochran's Q'' or related test procedures. This common procedure however is questionable for several reasons, namely, the low power of such tests especially in the very common case of only few estimates being combined in the analysis, as well as the specification of ''homogeneity'' as the null hypothesis which is then only rejected in the presence of sufficient evidence against it.

Estimation

While the main purpose of a meta-analysis usually is estimation of the ''main effect'', investigation of the ''heterogeneity'' is also crucial for its interpretation. A large number of ( frequentist and Bayesian) estimators is available. Bayesian estimation of the heterogeneity usually requires the specification of an appropriate prior distribution.

While many of these estimators behave similarly in case of a large number of studies, differences in particular arise in their behaviour in the common case of only few estimates. An incorrect zero between-study variance estimate is frequently obtained, leading to a false homogeneity assumption. Overall, it appears that heterogeneity is being consistently underestimated in meta-analyses.

Quantification

The heterogeneity ''variance'' is commonly denoted by τ², or the standard deviation (its square root) by τ. Heterogeneity is probably most readily interpretable in terms of τ, as this is the heterogeneity distribution's scale parameter, which is measured in the same units as the overall effect itself.

Another common measure of heterogeneity is I², a statistic that indicates the ''percentage of variance'' in a meta-analysis that is attributable to study heterogeneity (somewhat similarly to a coefficient of determination).
I² relates the heterogeneity variance's magnitude to the size of the individual estimates' variances (squared standard errors); with this normalisation however, it is not quite obvious what exactly would constitute "small" or "large" amounts of heterogeneity. For a constant heterogeneity (τ), the availability of smaller or larger studies (with correspondingly differing standard errors associated) would affect the I² measure; so the actual interpretation of an I² value is not straightforward.

The joint consideration of a ''prediction interval'' along with a confidence interval for the main effect may help getting a better sense of the contribution of heterogeneity to the uncertainty around the effect estimate.

See also

* Homogeneity (statistics)
* Random effects model
* Standard deviation, scale parameter, variance
* Meta-regression

References

Further reading

*
*
*
*
*

{{DEFAULTSORT:Study Heterogeneity
Systematic review
Meta-analysis