In
statistics
Statistics (from German language, German: ', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a s ...
, an effect size is a value measuring the strength of the relationship between two variables in a population, or a sample-based estimate of that quantity. It can refer to the value of a statistic calculated from a sample of
data
Data ( , ) are a collection of discrete or continuous values that convey information, describing the quantity, quality, fact, statistics, other basic units of meaning, or simply sequences of symbols that may be further interpreted for ...
, the value of one parameter for a hypothetical population, or to the equation that operationalizes how statistics or parameters lead to the effect size value.
Examples of effect sizes include the
correlation
In statistics, correlation or dependence is any statistical relationship, whether causal or not, between two random variables or bivariate data. Although in the broadest sense, "correlation" may indicate any type of association, in statistics ...
between two variables, the
regression coefficient in a regression, the
mean
A mean is a quantity representing the "center" of a collection of numbers and is intermediate to the extreme values of the set of numbers. There are several kinds of means (or "measures of central tendency") in mathematics, especially in statist ...
difference, or the risk of a particular event (such as a heart attack) happening. Effect sizes are a complement tool for
statistical hypothesis testing
A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. T ...
, and play an important role in
power analyses to assess the sample size required for new experiments. Effect size are fundamental in
meta-analyses
Meta-analysis is a method of synthesis of quantitative data from multiple independent studies addressing a common research question. An important part of this method involves computing a combined effect size across all of the studies. As such, th ...
which aim to provide the combined effect size based on data from multiple studies. The cluster of data-analysis methods concerning effect sizes is referred to as
estimation statistics
Estimation statistics, or simply estimation, is a data analysis framework that uses a combination of effect sizes, confidence intervals, precision planning, and meta-analysis to plan experiments, analyze data and interpret results. It complement ...
.
Effect size is an essential component when evaluating the strength of a statistical claim, and it is the first item (magnitude) in the
MAGIC criteria. The
standard deviation
In statistics, the standard deviation is a measure of the amount of variation of the values of a variable about its Expected value, mean. A low standard Deviation (statistics), deviation indicates that the values tend to be close to the mean ( ...
of the effect size is of critical importance, since it indicates how much uncertainty is included in the measurement. A standard deviation that is too large will make the measurement nearly meaningless. In meta-analysis, where the purpose is to combine multiple effect sizes, the uncertainty in the effect size is used to weigh effect sizes, so that large studies are considered more important than small studies. The uncertainty in the effect size is calculated differently for each type of effect size, but generally only requires knowing the study's sample size (''N''), or the number of observations (''n'') in each group.
Reporting effect sizes or estimates thereof (effect estimate
E estimate of effect) is considered good practice when presenting empirical research findings in many fields.
The reporting of effect sizes facilitates the interpretation of the importance of a research result, in contrast to its
statistical significance
In statistical hypothesis testing, a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true. More precisely, a study's defined significance level, denoted by \alpha, is the ...
.
Effect sizes are particularly prominent in
social science
Social science (often rendered in the plural as the social sciences) is one of the branches of science, devoted to the study of societies and the relationships among members within those societies. The term was formerly used to refer to the ...
and in
medical research
Medical research (or biomedical research), also known as health research, refers to the process of using scientific methods with the aim to produce knowledge about human diseases, the prevention and treatment of illness, and the promotion of ...
(where size of
treatment effect is important).
Effect sizes may be measured in relative or absolute terms. In relative effect sizes, two groups are directly compared with each other, as in
odds ratio
An odds ratio (OR) is a statistic that quantifies the strength of the association between two events, A and B. The odds ratio is defined as the ratio of the odds of event A taking place in the presence of B, and the odds of A in the absence of B ...
s and
relative risk
The relative risk (RR) or risk ratio is the ratio of the probability of an outcome in an exposed group to the probability of an outcome in an unexposed group. Together with risk difference and odds ratio, relative risk measures the association bet ...
s. For absolute effect sizes, a larger
absolute value
In mathematics, the absolute value or modulus of a real number x, is the non-negative value without regard to its sign. Namely, , x, =x if x is a positive number, and , x, =-x if x is negative (in which case negating x makes -x positive), ...
always indicates a stronger effect. Many types of measurements can be expressed as either absolute or relative, and these can be used together because they convey different information. A prominent task force in the psychology research community made the following recommendation:
Overview
Population and sample effect sizes
As in
statistical estimation
Estimation theory is a branch of statistics that deals with estimating the values of parameters based on measured empirical data that has a random component. The parameters describe an underlying physical setting in such a way that their value ...
, the true effect size is distinguished from the observed effect size. For example, to measure the risk of disease in a population (the population effect size) one can measure the risk within a sample of that population (the sample effect size). Conventions for describing true and observed effect sizes follow standard statistical practices—one common approach is to use Greek letters like ρ
hoto denote population parameters and Latin letters like ''r'' to denote the corresponding statistic. Alternatively, a "hat" can be placed over the population parameter to denote the statistic, e.g. with
being the estimate of the parameter
.
As in any statistical setting, effect sizes are estimated with
sampling error
In statistics, sampling errors are incurred when the statistical characteristics of a population are estimated from a subset, or sample, of that population. Since the sample does not include all members of the population, statistics of the sample ...
, and may be biased unless the effect size estimator that is used is appropriate for the manner in which the data were
sampled and the manner in which the measurements were made. An example of this is
publication bias
In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a Statistical significance, significant find ...
, which occurs when scientists report results only when the estimated effect sizes are large or are statistically significant. As a result, if many researchers carry out studies with low statistical power, the reported effect sizes will tend to be larger than the true (population) effects, if any.
Another example where effect sizes may be distorted is in a multiple-trial experiment, where the effect size calculation is based on the averaged or aggregated response across the trials.
Smaller studies sometimes show different, often larger, effect sizes than larger studies. This phenomenon is known as the small-study effect, which may signal publication bias.
Relationship to test statistics
Sample-based effect sizes are distinguished from
test statistic
Test statistic is a quantity derived from the sample for statistical hypothesis testing.Berger, R. L.; Casella, G. (2001). ''Statistical Inference'', Duxbury Press, Second Edition (p.374) A hypothesis test is typically specified in terms of a tes ...
s used in hypothesis testing, in that they estimate the strength (magnitude) of, for example, an apparent relationship, rather than assigning a
significance level reflecting whether the magnitude of the relationship observed could be due to chance. The effect size does not directly determine the significance level, or vice versa. Given a sufficiently large sample size, a non-null statistical comparison will always show a statistically significant result unless the population effect size is exactly zero (and even there it will show statistical significance at the rate of the Type I error used). For example, a sample
Pearson correlation coefficient of 0.01 is statistically significant if the sample size is 1000. Reporting only the significant
''p''-value from this analysis could be misleading if a correlation of 0.01 is too small to be of interest in a particular application.
Standardized and unstandardized effect sizes
The term ''effect size'' can refer to a standardized measure of effect (such as ''r'',
Cohen's ''d'', or the
odds ratio
An odds ratio (OR) is a statistic that quantifies the strength of the association between two events, A and B. The odds ratio is defined as the ratio of the odds of event A taking place in the presence of B, and the odds of A in the absence of B ...
), or to an unstandardized measure (e.g., the difference between group means or the unstandardized regression coefficients). Standardized effect size measures are typically used when:
* the metrics of variables being studied do not have intrinsic meaning (e.g., a score on a personality test on an arbitrary scale),
* results from multiple studies are being combined,
* some or all of the studies use different scales, or
* it is desired to convey the size of an effect relative to the variability in the population.
In meta-analyses, standardized effect sizes are used as a common measure that can be calculated for different studies and then combined into an overall summary.
Interpretation
The interpretation of an effect size of being ''small'', ''medium'', or ''large'' depends on its substantive context and its operational definition. Jacob Cohen
suggested interpretation guidelines that are near ubiquitous across many fields. However, Cohen also cautioned:
"The terms 'small,' 'medium,' and 'large' are relative, not only to each other, but to the area of behavioral science or even more particularly to the specific content and research method being employed in any given investigation... In the face of this relativity, there is a certain risk inherent in offering conventional operational definitions for these terms for use in power analysis in as diverse a field of inquiry as behavioral science. This risk is nevertheless accepted in the belief that more is to be gained than lost by supplying a common conventional frame of reference which is recommended for use only when no better basis for estimating the ES index is available." (p. 25)
Sawilowsky
[ http://digitalcommons.wayne.edu/jmasm/vol8/iss2/26/] recommended that the rules of thumb for effect sizes should be revised, and expanded the descriptions to include ''very small'', ''very large'', and ''huge''. Funder and Ozer
suggested that effect sizes should be interpreted based on benchmarks and consequences of findings, resulting in adjustment of guideline recommendations.
noted for a ''medium'' effect size, "you'll choose the same ''n'' regardless of the accuracy or reliability of your instrument, or the narrowness or diversity of your subjects. Clearly, important considerations are being ignored here. Researchers should interpret the substantive significance of their results by grounding them in a meaningful context or by quantifying their contribution to knowledge, and Cohen's effect size descriptions can be helpful as a starting point."
Similarly, a U.S. Dept of Education sponsored report argued that the widespread indiscriminate use of Cohen's interpretation guidelines can be inappropriate and misleading.
They instead suggested that norms should be based on distributions of effect sizes from comparable studies. Thus a small effect (in absolute numbers) could be considered ''large'' if the effect is larger than similar studies in the field. See
Abelson's paradox and Sawilowsky's paradox for related points.
The table below contains descriptors for various magnitudes of ''d'', ''r'', ''f'' and ''omega'', as initially suggested by Jacob Cohen,
and later expanded by Sawilowsky,
and by Funder & Ozer.
Types
About 50 to 100 different measures of effect size are known. Many effect sizes of different types can be converted to other types, as many estimate the separation of two distributions, so are mathematically related. For example, a correlation coefficient can be converted to a Cohen's d and vice versa.
Correlation family: Effect sizes based on "variance explained"
These effect sizes estimate the amount of the variance within an experiment that is "explained" or "accounted for" by the experiment's model (
Explained variation).
Pearson ''r'' or correlation coefficient
Pearson's correlation, often denoted ''r'' and introduced by
Karl Pearson
Karl Pearson (; born Carl Pearson; 27 March 1857 – 27 April 1936) was an English biostatistician and mathematician. He has been credited with establishing the discipline of mathematical statistics. He founded the world's first university ...
, is widely used as an ''effect size'' when paired quantitative data are available; for instance if one were studying the relationship between birth weight and longevity. The correlation coefficient can also be used when the data are binary. Pearson's ''r'' can vary in magnitude from −1 to 1, with −1 indicating a perfect negative linear relation, 1 indicating a perfect positive linear relation, and 0 indicating no linear relation between two variables.
= Coefficient of determination (''r''2 or ''R''2)
=
A related ''effect size'' is ''r''
2, the
coefficient of determination (also referred to as ''R''
2 or "''r''-squared"), calculated as the square of the Pearson correlation ''r''. In the case of paired data, this is a measure of the proportion of variance shared by the two variables, and varies from 0 to 1. For example, with an ''r'' of 0.21 the coefficient of determination is 0.0441, meaning that 4.4% of the variance of either variable is shared with the other variable. The ''r''
2 is always positive, so does not convey the direction of the correlation between the two variables.
= Eta-squared (''η''2)
=
Eta-squared describes the ratio of variance explained in the dependent variable by a predictor while controlling for other predictors, making it analogous to the ''r''
2. Eta-squared is a biased estimator of the variance explained by the model in the population (it estimates only the effect size in the sample). This estimate shares the weakness with ''r''
2 that each additional variable will automatically increase the value of ''η''
2. In addition, it measures the variance explained of the sample, not the population, meaning that it will always overestimate the effect size, although the bias grows smaller as the sample grows larger.
= Omega-squared (''ω''2)
=
A less biased estimator of the variance explained in the population is ''ω''
2[Tabachnick, B.G. & Fidell, L.S. (2007). Chapter 4: "Cleaning up your act. Screening data prior to analysis", p. 55 In B.G. Tabachnick & L.S. Fidell (Eds.), ''Using Multivariate Statistics'', Fifth Edition. Boston: Pearson Education, Inc. / Allyn and Bacon.]
This form of the formula is limited to between-subjects analysis with equal sample sizes in all cells.
Since it is less biased (although not ''un''biased), ''ω''
2 is preferable to η
2; however, it can be more inconvenient to calculate for complex analyses. A generalized form of the estimator has been published for between-subjects and within-subjects analysis, repeated measure, mixed design, and randomized block design experiments.
In addition, methods to calculate partial ''ω''
2 for individual factors and combined factors in designs with up to three independent variables have been published.
[
]
Cohen's ''f''2
Cohen's ''f''2 is one of several effect size measures to use in the context of an F-test
An F-test is a statistical test that compares variances. It is used to determine if the variances of two samples, or if the ratios of variances among multiple samples, are significantly different. The test calculates a Test statistic, statistic, ...
for ANOVA or multiple regression. Its amount of bias (overestimation of the effect size for the ANOVA) depends on the bias of its underlying measurement of variance explained (e.g., ''R''2, ''η''2, ''ω''2).
The ''f''2 effect size measure for multiple regression is defined as:
Likewise, ''f''2 can be defined as:
or
for models described by those effect size measures.
The effect size measure for sequential multiple regression and also common for PLS modeling is defined as:
where ''R''2''A'' is the variance accounted for by a set of one or more independent variables ''A'', and ''R''2''AB'' is the combined variance accounted for by ''A'' and another set of one or more independent variables of interest ''B''. By convention, ''f''2 effect sizes of , , and are termed ''small'', ''medium'', and ''large'', respectively.
Cohen's can also be found for factorial analysis of variance (ANOVA) working backwards, using:
In a balanced design (equivalent sample sizes across groups) of ANOVA, the corresponding population parameter of is
wherein ''μ''''j'' denotes the population mean within the ''j''th group of the total ''K'' groups, and ''σ'' the equivalent population standard deviations within each groups. ''SS'' is the sum of squares in ANOVA.
Cohen's ''q''
Another measure that is used with correlation differences is Cohen's q. This is the difference between two Fisher transformed Pearson regression coefficients. In symbols this is
where ''r''1 and ''r''2 are the regressions being compared. The expected value of ''q'' is zero and its variance is
where ''N''1 and ''N''2 are the number of data points in the first and second regression respectively.
Difference family: Effect sizes based on differences between means
The raw effect size pertaining to a comparison of two groups is inherently calculated as the differences between the two means. However, to facilitate interpretation it is common to standardise the effect size; various conventions for statistical standardisation are presented below.
Standardized mean difference
A (population) effect size ''θ'' based on means usually considers the standardized mean difference (SMD) between two populations
where ''μ''1 is the mean for one population, ''μ''2 is the mean for the other population, and σ is a standard deviation
In statistics, the standard deviation is a measure of the amount of variation of the values of a variable about its Expected value, mean. A low standard Deviation (statistics), deviation indicates that the values tend to be close to the mean ( ...
based on either or both populations.
In the practical setting the population values are typically not known and must be estimated from sample statistics. The several versions of effect sizes based on means differ with respect to which statistics are used.
This form for the effect size resembles the computation for a ''t''-test statistic, with the critical difference that the ''t''-test statistic includes a factor of . This means that for a given effect size, the significance level increases with the sample size. Unlike the ''t''-test statistic, the effect size aims to estimate a population parameter
A parameter (), generally, is any characteristic that can help in defining or classifying a particular system (meaning an event, project, object, situation, etc.). That is, a parameter is an element of a system that is useful, or critical, when ...
and is not affected by the sample size.
SMD values of 0.2 to 0.5 are considered small, 0.5 to 0.8 are considered medium, and greater than 0.8 are considered large.
Cohen's ''d''
Cohen's ''d'' is defined as the difference between two means divided by a standard deviation for the data, i.e.
Jacob Cohen defined ''s'', the pooled standard deviation, as (for two independent samples):
where the variance for one of the groups is defined as
and similarly for the other group.
Other authors choose a slightly different computation of the standard deviation when referring to "Cohen's ''d''" where the denominator is without "-2"
This definition of "Cohen's ''d''" is termed the maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed stati ...
estimator by Hedges and Olkin,
and it is related to Hedges' ''g'' by a scaling factor (see below).
With two paired samples, an approach is to look at the distribution of the difference scores. In that case, ''s'' is the standard deviation of this distribution of difference scores (of note, the standard deviation of difference scores is dependent on the correlation between paired samples). This creates the following relationship between the t-statistic to test for a difference in the means of the two paired groups and Cohen's ''d (computed with difference scores):
and
However, for paired samples, Cohen states that d' does not provide the correct estimate to obtain the power of the test for d, and that before looking the values up in the tables provided for d, it should be corrected for r as in the following formula:
where r is the correlation between paired measurements. Given the same sample size, the higher r, the higher the power for a test of paired difference.
Since d' depends on r, as a measure of effect size it is difficult to interpret; therefore, in the context of paired analyses, since it is possible to compute d' or d (estimated with a pooled standard deviation or that of a group or time-point), it is necessary to explicitly indicate which one is being reported. As a measure of effect size, d (estimated with a pooled standard deviation or that of a group or time-point) is more appropriate, for instance in meta-analysis.[ ]
Cohen's ''d'' is frequently used in estimating sample sizes for statistical testing. A lower Cohen's ''d'' indicates the necessity of larger sample sizes, and vice versa, as can subsequently be determined together with the additional parameters of desired significance level and statistical power
In frequentist statistics, power is the probability of detecting a given effect (if that effect actually exists) using a given test in a given context. In typical use, it is a function of the specific test that is used (including the choice of tes ...
.
Glass' Δ
In 1976, Gene V. Glass proposed an estimator of the effect size that uses only the standard deviation of the second group
The second group may be regarded as a control group, and Glass argued that if several treatments were compared to the control group it would be better to use just the standard deviation computed from the control group, so that effect sizes would not differ under equal means and different variances.
Under a correct assumption of equal population variances a pooled estimate for ''σ'' is more precise.
Hedges' ''g''
Hedges' ''g'', suggested by Larry Hedges in 1981,
is like the other measures based on a standardized difference
where the pooled standard deviation is computed as:
However, as an estimator
In statistics, an estimator is a rule for calculating an estimate of a given quantity based on Sample (statistics), observed data: thus the rule (the estimator), the quantity of interest (the estimand) and its result (the estimate) are distinguish ...
for the population effect size ''θ'' it is bias
Bias is a disproportionate weight ''in favor of'' or ''against'' an idea or thing, usually in a way that is inaccurate, closed-minded, prejudicial, or unfair. Biases can be innate or learned. People may develop biases for or against an individ ...
ed.
Nevertheless, this bias can be approximately corrected through multiplication by a factor
Hedges and Olkin refer to this less-biased estimator as ''d'', but it is not the same as Cohen's ''d''.
The exact form for the correction factor ''J''() involves the gamma function
In mathematics, the gamma function (represented by Γ, capital Greek alphabet, Greek letter gamma) is the most common extension of the factorial function to complex numbers. Derived by Daniel Bernoulli, the gamma function \Gamma(z) is defined ...
There are also multilevel variants of Hedges' g, e.g., for use in cluster randomised controlled trials (CRTs). CRTs involve randomising clusters, such as schools or classrooms, to different conditions and are frequently used in education research.
Ψ, root-mean-square standardized effect
A similar effect size estimator for multiple comparisons (e.g., ANOVA) is the Ψ root-mean-square standardized effect:
where ''k'' is the number of groups in the comparisons.
This essentially presents the omnibus difference of the entire model adjusted by the root mean square, analogous to ''d'' or ''g''.
In addition, a generalization for multi-factorial designs has been provided.
Distribution of effect sizes based on means
Provided that the data is Gaussian distributed a scaled Hedges' ''g'', , follows a noncentral ''t''-distribution with the noncentrality parameter and degrees of freedom. Likewise, the scaled Glass' Δ is distributed with degrees of freedom.
From the distribution it is possible to compute the expectation and variance of the effect sizes.
In some cases large sample approximations for the variance are used. One suggestion for the variance of Hedges' unbiased estimator is
Strictly standardized mean difference (SSMD)
As a statistical parameter, SSMD (denoted as ) is defined as the ratio of mean
A mean is a quantity representing the "center" of a collection of numbers and is intermediate to the extreme values of the set of numbers. There are several kinds of means (or "measures of central tendency") in mathematics, especially in statist ...
to standard deviation
In statistics, the standard deviation is a measure of the amount of variation of the values of a variable about its Expected value, mean. A low standard Deviation (statistics), deviation indicates that the values tend to be close to the mean ( ...
of the difference of two random values respectively from two groups. Assume that one group with random values has mean
A mean is a quantity representing the "center" of a collection of numbers and is intermediate to the extreme values of the set of numbers. There are several kinds of means (or "measures of central tendency") in mathematics, especially in statist ...
and variance
In probability theory and statistics, variance is the expected value of the squared deviation from the mean of a random variable. The standard deviation (SD) is obtained as the square root of the variance. Variance is a measure of dispersion ...
and another group has mean
A mean is a quantity representing the "center" of a collection of numbers and is intermediate to the extreme values of the set of numbers. There are several kinds of means (or "measures of central tendency") in mathematics, especially in statist ...
and variance
In probability theory and statistics, variance is the expected value of the squared deviation from the mean of a random variable. The standard deviation (SD) is obtained as the square root of the variance. Variance is a measure of dispersion ...
. The covariance
In probability theory and statistics, covariance is a measure of the joint variability of two random variables.
The sign of the covariance, therefore, shows the tendency in the linear relationship between the variables. If greater values of one ...
between the two groups is Then, the SSMD for the comparison of these two groups is defined as
:
If the two groups are independent,
:
If the two independent groups have equal variance
In probability theory and statistics, variance is the expected value of the squared deviation from the mean of a random variable. The standard deviation (SD) is obtained as the square root of the variance. Variance is a measure of dispersion ...
s ,
:
Other metrics
Mahalanobis distance
The Mahalanobis distance is a distance measure, measure of the distance between a point P and a probability distribution D, introduced by Prasanta Chandra Mahalanobis, P. C. Mahalanobis in 1936. The mathematical details of Mahalanobis distance ...
(D) is a multivariate generalization of Cohen's d, which takes into account the relationships between the variables.
Categorical family: Effect sizes for associations among categorical variables
Commonly used measures of association for the chi-squared test
A chi-squared test (also chi-square or test) is a Statistical hypothesis testing, statistical hypothesis test used in the analysis of contingency tables when the sample sizes are large. In simpler terms, this test is primarily used to examine w ...
are the Phi coefficient and Cramér's V (sometimes referred to as Cramér's phi and denoted as ''φ''''c''). Phi is related to the point-biserial correlation coefficient
The point biserial correlation coefficient (''rpb'') is a correlation coefficient used when one variable (e.g. ''Y'') is dichotomous; ''Y'' can either be "naturally" dichotomous, like whether a coin lands heads or tails, or an artificially dichot ...
and Cohen's ''d'' and estimates the extent of the relationship between two variables (2 × 2).[Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November)]
Equating r-based and d-based effect-size indices: Problems with a commonly recommended formula.
Paper presented at the annual meeting of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353) Cramér's V may be used with variables having more than two levels.
Phi can be computed by finding the square root of the chi-squared statistic divided by the sample size.
Similarly, Cramér's V is computed by taking the square root of the chi-squared statistic divided by the sample size and the length of the minimum dimension (''k'' is the smaller of the number of rows ''r'' or columns ''c'').
φ''c'' is the intercorrelation of the two discrete variables and may be computed for any value of ''r'' or ''c''. However, as chi-squared values tend to increase with the number of cells, the greater the difference between ''r'' and ''c'', the more likely V will tend to 1 without strong evidence of a meaningful correlation.
Cohen's omega (''ω'')
Another measure of effect size used for chi-squared tests is Cohen's omega (). This is defined as
where ''p''0''i'' is the proportion of the ''i''th cell under ''H''0, ''p''1''i'' is the proportion of the ''i''th cell under ''H''1 and ''m'' is the number of cells.
Odds ratio
The odds ratio
An odds ratio (OR) is a statistic that quantifies the strength of the association between two events, A and B. The odds ratio is defined as the ratio of the odds of event A taking place in the presence of B, and the odds of A in the absence of B ...
(OR) is another useful effect size. It is appropriate when the research question focuses on the degree of association between two binary variables. For example, consider a study of spelling ability. In a control group, two students pass the class for every one who fails, so the odds of passing are two to one (or 2/1 = 2). In the treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. Odds ratio statistics are on a different scale than Cohen's ''d'', so this '3' is not comparable to a Cohen's ''d'' of 3.
Relative risk
The relative risk
The relative risk (RR) or risk ratio is the ratio of the probability of an outcome in an exposed group to the probability of an outcome in an unexposed group. Together with risk difference and odds ratio, relative risk measures the association bet ...
(RR), also called risk ratio, is simply the risk (probability) of an event relative to some independent variable. This measure of effect size differs from the odds ratio in that it compares ''probabilities'' instead of ''odds'', but asymptotically approaches the latter for small probabilities. Using the example above, the ''probabilities'' for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively. The effect size can be computed the same as above, but using the probabilities instead. Therefore, the relative risk is 1.28. Since rather large probabilities of passing were used, there is a large difference between relative risk and odds ratio. Had ''failure'' (a smaller probability) been used as the event (rather than ''passing''), the difference between the two measures of effect size would not be so great.
While both measures are useful, they have different statistical uses. In medical research, the odds ratio
An odds ratio (OR) is a statistic that quantifies the strength of the association between two events, A and B. The odds ratio is defined as the ratio of the odds of event A taking place in the presence of B, and the odds of A in the absence of B ...
is commonly used for case-control studies, as odds, but not probabilities, are usually estimated. Relative risk is commonly used in randomized controlled trial
A randomized controlled trial (or randomized control trial; RCT) is a form of scientific experiment used to control factors not under direct experimental control. Examples of RCTs are clinical trials that compare the effects of drugs, surgical ...
s and cohort studies
A cohort study is a particular form of longitudinal study that samples a cohort (a group of people who share a defining characteristic, typically those who experienced a common event in a selected period, such as birth or graduation), performing ...
, but relative risk contributes to overestimations of the effectiveness of interventions.
Risk difference
The risk difference (RD), sometimes called absolute risk reduction, is simply the difference in risk (probability) of an event between two groups. It is a useful measure in experimental research, since RD tells you the extent to which an experimental interventions changes the probability of an event or outcome. Using the example above, the probabilities for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively, and so the RD effect size is 0.86 − 0.67 = 0.19 (or 19%). RD is the superior measure for assessing effectiveness of interventions.
Cohen's ''h''
One measure used in power analysis when comparing two independent proportions is Cohen's ''h''. This is defined as follows
where ''p''1 and ''p''2 are the proportions of the two samples being compared and arcsin is the arcsine transformation.
Probability of superiority
To more easily describe the meaning of an effect size to people outside statistics, the common language effect size, as the name implies, was designed to communicate it in plain English. It is used to describe a difference between two groups and was proposed, as well as named, by Kenneth McGraw and S. P. Wong in 1992. They used the following example (about heights of men and women): "in any random pairing of young adult males and females, the probability of the male being taller than the female is .92, or in simpler terms yet, in 92 out of 100 blind dates among young adults, the male will be taller than the female", when describing the population value of the common language effect size.
Effect size for ordinal data
Cliff's delta or , originally developed by Norman Cliff
Norman Cliff (born September 1, 1930) is an American psychologist. He received his Ph.D. from Princeton University, Princeton in psychometrics in 1957. After research positions in the United States Public Health Service, US Public Health Service ...
for use with ordinal data, is a measure of how often the values in one distribution are larger than the values in a second distribution. Crucially, it does not require any assumptions about the shape or spread of the two distributions.
The sample estimate is given by:
where the two distributions are of size and with items and , respectively, and