Combinatorial meta-analysis (CMA) is the study of the behaviour of statistical properties of combinations of studies from a meta-analytic dataset (typically in social science research). In an article that develops the notion of "gravity" in the context of meta-analysis, Travis GeeGee, T. (2005) "Capturing study influence: The concept of 'gravity' in meta-analysis", ''Counselling, Psychotherapy, and Health'', 1(1), 52–7

proposed that the jackknife methods applied to

meta-analysis Meta-analysis is a method of synthesis of quantitative data from multiple independent studies addressing a common research question. An important part of this method involves computing a combined effect size across all of the studies. As such, th ...

in that article could be extended to examine all possible combinations of studies (where practical) or random subsets of studies (where the

combinatorics Combinatorics is an area of mathematics primarily concerned with counting, both as a means and as an end to obtaining results, and certain properties of finite structures. It is closely related to many other areas of mathematics and has many ...

of the situation made it computationally infeasible).

Concept

In the original article, ''k'' objects (studies) are combined ''k''-1 at a time ( jackknife estimation), resulting in ''k'' estimates. It is observed that this is a special case of the more general approach of CMA which computes results for ''k'' studies taken 1, 2, 3 ... ''k'' − 1, ''k'' at a time. Where it is computationally feasible to obtain all possible combinations, the resulting distribution of statistics is termed "exact CMA." Where the number of possible combinations is prohibitively large, it is termed "approximate CMA." CMA makes it possible to study the relative behaviour of different statistics under combinatorial conditions. This differs from the standard approach in

of adopting a single method and computing a single result, and allows significant triangulation to occur, by computing different indices for each combination and examining whether they all tell the same story.

Implications

An implication of this is that where multiple random intercepts exist, the

heterogeneity Homogeneity and heterogeneity are concepts relating to the uniformity of a substance, process or image. A homogeneous feature is uniform in composition or character (i.e., color, shape, size, weight, height, distribution, texture, language, i ...

within certain combinations will be minimized. CMA can thus be used as a

data mining Data mining is the process of extracting and finding patterns in massive data sets involving methods at the intersection of machine learning, statistics, and database systems. Data mining is an interdisciplinary subfield of computer science and ...

method to identify the number of intercepts that may be present in the dataset by looking at which studies are included in the local minima that may be obtained through recombination. A further implication of this is that arguments over inclusion or exclusion of studies may be moot when the distribution of all possible results is taken into account. A useful tool developed by Gee (reference to come when published) is the "PPES" plot (standing for "Probability of Positive Effect Size", assuming differences are scaled such that larger in a positive direction is desired). For each subset of combinations, where studies are taken ''j'' = 1, 2, ... ''k'' − 1, ''k'' at a time, the proportion of results that show a positive effect size (either WMD or SMD will work) is taken, and this is plotted against ''j''. This can be adapted to a "PMES" plot (standing for "Probability of Minimal Effect Size"), where the proportion of studies exceeding some minimal effect size (e.g., SMD = 0.10) is taken for each value of ''j'' = 1, 2, ... ''k'' − 1, ''k''. Where a clear effect is present, this plot should asymptote to near 1.0 fairly rapidly. With this, it is possible then that, for instance, disputes over the inclusion or exclusion of two or three studies out of a dozen or more may be framed in the context of a plot that shows a clear effect for ''any'' combination of 7 or more studies. It is also possible through CMA to examine the relationship of covariates with effect sizes. For example, if industry funding is suspected as a source of bias, then the proportion of studies in a given subset that were industry funded can be computed and plotted directly against the effect size estimate. If average age in the various studies was itself fairly variable, then the mean of these means across studies in a given combination can be obtained, and similarly plotted.

Implementations

Gee's original software for performing jackknife and combinatorial meta analysis was based on older meta-analytic macros written in the SAS programming language. It was the basis of one report in the area of arthritis treatment. While this software was shared with colleagues informally, it was not published. A later meta-analysis applied the concept in the context of the treatment of diarrhea. A jackknife method was applied to meta-analytic data some years later but it does not appear that specialized software was developed for the task. Other commentators have also called for related methods, apparently unaware of the original work. More recent work by a software porting team at Brown University has implemented the concept in STATA.

Limitations

CMA does not solve

's problem of "garbage in, garbage out". However, when a class of studies is ''deemed'' garbage by a critic, it does offer a way of examining the extent to which those studies may have changed a result. Similarly, it offers no direct solution to the problem of which method to choose for combination or weighting. What it does offer, as noted above, is triangulation, where agreements between methods may be obtained, and disagreements between methods understood across the range of possible combinations of studies.

References

{{Reflist Meta-analysis