Linear discriminant analysis (LDA), normal discriminant analysis (NDA), or discriminant function analysis is a generalization of Fisher's linear discriminant, a method used in
statistics
Statistics (from German language, German: ''wikt:Statistik#German, Statistik'', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of ...
and other fields, to find a
linear combination of features that characterizes or separates two or more classes of objects or events. The resulting combination may be used as a
linear classifier
In the field of machine learning, the goal of statistical classification is to use an object's characteristics to identify which class (or group) it belongs to. A linear classifier achieves this by making a classification decision based on the val ...
, or, more commonly, for
dimensionality reduction
Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally ...
before later
classification Classification is a process related to categorization, the process in which ideas and objects are recognized, differentiated and understood.
Classification is the grouping of related facts into classes.
It may also refer to:
Business, organizat ...
.
LDA is closely related to
analysis of variance
Analysis of variance (ANOVA) is a collection of statistical models and their associated estimation procedures (such as the "variation" among and between groups) used to analyze the differences among means. ANOVA was developed by the statisticia ...
(ANOVA) and
regression analysis
In statistical modeling, regression analysis is a set of statistical processes for estimating the relationships between a dependent variable (often called the 'outcome' or 'response' variable, or a 'label' in machine learning parlance) and one ...
, which also attempt to express one
dependent variable
Dependent and independent variables are variables in mathematical modeling, statistical modeling and experimental sciences. Dependent variables receive this name because, in an experiment, their values are studied under the supposition or demand ...
as a linear combination of other features or measurements.
[
] However, ANOVA uses
categorical independent variables
Dependent and independent variables are variables in mathematical modeling, statistical modeling and experimental sciences. Dependent variables receive this name because, in an experiment, their values are studied under the supposition or deman ...
and a
continuous
Continuity or continuous may refer to:
Mathematics
* Continuity (mathematics), the opposing concept to discreteness; common examples include
** Continuous probability distribution or random variable in probability and statistics
** Continuous ...
dependent variable
Dependent and independent variables are variables in mathematical modeling, statistical modeling and experimental sciences. Dependent variables receive this name because, in an experiment, their values are studied under the supposition or demand ...
, whereas discriminant analysis has continuous
independent variables
Dependent and independent variables are variables in mathematical modeling, statistical modeling and experimental sciences. Dependent variables receive this name because, in an experiment, their values are studied under the supposition or deman ...
and a categorical dependent variable (''i.e.'' the class label).
Logistic regression
In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear function (calculus), linear combination of one or more independent var ...
and
probit regression
In statistics, a probit model is a type of regression where the dependent variable can take only two values, for example married or not married. The word is a portmanteau, coming from ''probability'' + ''unit''. The purpose of the model is to e ...
are more similar to LDA than ANOVA is, as they also explain a categorical variable by the values of continuous independent variables. These other methods are preferable in applications where it is not reasonable to assume that the independent variables are normally distributed, which is a fundamental assumption of the LDA method.
LDA is also closely related to
principal component analysis
Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and ...
(PCA) and
factor analysis
Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. For example, it is possible that variations in six observed ...
in that they both look for linear combinations of variables which best explain the data.
LDA explicitly attempts to model the difference between the classes of data. PCA, in contrast, does not take into account any difference in class, and factor analysis builds the feature combinations based on differences rather than similarities. Discriminant analysis is also different from factor analysis in that it is not an interdependence technique: a distinction between independent variables and dependent variables (also called criterion variables) must be made.
LDA works when the measurements made on independent variables for each observation are continuous quantities. When dealing with categorical independent variables, the equivalent technique is discriminant correspondence analysis.
[Abdi, H. (2007]
"Discriminant correspondence analysis."
In: N.J. Salkind (Ed.): ''Encyclopedia of Measurement and Statistic''. Thousand Oaks (CA): Sage. pp. 270–275.
Discriminant analysis is used when groups are known a priori (unlike in
cluster analysis
Cluster analysis or clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar (in some sense) to each other than to those in other groups (clusters). It is a main task of ...
). Each case must have a score on one or more quantitative predictor measures, and a score on a group measure.
[Büyüköztürk, Ş. & Çokluk-Bökeoğlu, Ö. (2008)]
Discriminant function analysis: Concept and application
Egitim Arastirmalari - Eurasian Journal of Educational Research, 33, 73-92. In simple terms, discriminant function analysis is classification - the act of distributing things into groups, classes or categories of the same type.
History
The original
dichotomous
A dichotomy is a partition of a whole (or a set) into two parts (subsets). In other words, this couple of parts must be
* jointly exhaustive: everything must belong to one part or the other, and
* mutually exclusive: nothing can belong simult ...
discriminant analysis was developed by Sir
Ronald Fisher
Sir Ronald Aylmer Fisher (17 February 1890 – 29 July 1962) was a British polymath who was active as a mathematician, statistician, biologist, geneticist, and academic. For his work in statistics, he has been described as "a genius who a ...
in 1936.
[Cohen et al. Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences 3rd ed. (2003). Taylor & Francis Group.] It is different from an
ANOVA
Analysis of variance (ANOVA) is a collection of statistical models and their associated estimation procedures (such as the "variation" among and between groups) used to analyze the differences among means. ANOVA was developed by the statistician ...
or
MANOVA
In statistics, multivariate analysis of variance (MANOVA) is a procedure for comparing multivariate sample means. As a multivariate procedure, it is used when there are two or more dependent variables, and is often followed by significance tests ...
, which is used to predict one (ANOVA) or multiple (MANOVA) continuous dependent variables by one or more independent categorical variables. Discriminant function analysis is useful in determining whether a set of variables is effective in predicting category membership.
[Green, S.B. Salkind, N. J. & Akey, T. M. (2008)]
Using SPSS for Windows and Macintosh: Analyzing and understanding data
New Jersey: Prentice Hall.
LDA for two classes
Consider a set of observations
(also called features, attributes, variables or measurements) for each sample of an object or event with known class
. This set of samples is called the
training set
In machine learning, a common task is the study and construction of algorithms that can learn from and make predictions on data. Such algorithms function by making data-driven predictions or decisions, through building a mathematical model from ...
. The classification problem is then to find a good predictor for the class
of any sample of the same distribution (not necessarily from the training set) given only an observation
.
[
]
LDA approaches the problem by assuming that the conditional
probability density function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) can ...
s
and
are both
the normal distribution
''The'' () is a grammatical article in English, denoting persons or things already mentioned, under discussion, implied or otherwise presumed familiar to listeners, readers, or speakers. It is the definite article in English. ''The'' is the ...
with mean and
covariance
In probability theory and statistics, covariance is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the ...
parameters
and
, respectively. Under this assumption, the
Bayes optimal solution is to predict points as being from the second class if the log of the likelihood ratios is bigger than some threshold T, so that:
:
Without any further assumptions, the resulting classifier is referred to as
quadratic discriminant analysis (QDA).
LDA instead makes the additional simplifying
homoscedastic
In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. The s ...
ity assumption (''i.e.'' that the class covariances are identical, so
) and that the covariances have full rank.
In this case, several terms cancel:
:
:
because
is
Hermitian {{Short description, none
Numerous things are named after the French mathematician Charles Hermite (1822–1901):
Hermite
* Cubic Hermite spline, a type of third-degree spline
* Gauss–Hermite quadrature, an extension of Gaussian quadrature m ...
and the above decision criterion
becomes a threshold on the
dot product
In mathematics, the dot product or scalar productThe term ''scalar product'' means literally "product with a scalar as a result". It is also used sometimes for other symmetric bilinear forms, for example in a pseudo-Euclidean space. is an algebra ...
:
for some threshold constant ''c'', where
:
:
This means that the criterion of an input
being in a class
is purely a function of this linear combination of the known observations.
It is often useful to see this conclusion in geometrical terms: the criterion of an input
being in a class
is purely a function of projection of multidimensional-space point
onto vector
(thus, we only consider its direction). In other words, the observation belongs to
if corresponding
is located on a certain side of a hyperplane perpendicular to
. The location of the plane is defined by the threshold c.
Assumptions
The assumptions of discriminant analysis are the same as those for MANOVA. The analysis is quite sensitive to outliers and the size of the smallest group must be larger than the number of predictor variables.
*
Multivariate normality: Independent variables are normal for each level of the grouping variable.
*Homogeneity of variance/covariance (
homoscedasticity
In statistics, a sequence (or a vector) of random variables is homoscedastic () if all its random variables have the same finite variance. This is also known as homogeneity of variance. The complementary notion is called heteroscedasticity. The s ...
): Variances among group variables are the same across levels of predictors. Can be tested with
Box's M statistic.
It has been suggested, however, that linear discriminant analysis be used when covariances are equal, and that
quadratic discriminant analysis may be used when covariances are not equal.
*
Multicollinearity
In statistics, multicollinearity (also collinearity) is a phenomenon in which one predictor variable in a multiple regression model can be linearly predicted from the others with a substantial degree of accuracy. In this situation, the coeffic ...
: Predictive power can decrease with an increased correlation between predictor variables.
*
Independence
Independence is a condition of a person, nation, country, or state in which residents and population, or some portion thereof, exercise self-government, and usually sovereignty, over its territory. The opposite of independence is the statu ...
: Participants are assumed to be randomly sampled, and a participant's score on one variable is assumed to be independent of scores on that variable for all other participants.
It has been suggested that discriminant analysis is relatively robust to slight violations of these assumptions, and it has also been shown that discriminant analysis may still be reliable when using dichotomous variables (where multivariate normality is often violated).
Discriminant functions
Discriminant analysis works by creating one or more linear combinations of predictors, creating a new
latent variable
In statistics, latent variables (from Latin: present participle of ''lateo'', “lie hidden”) are variables that can only be inferred indirectly through a mathematical model from other observable variables that can be directly observed or me ...
for each function. These functions are called discriminant functions. The number of functions possible is either
where
= number of groups, or
(the number of predictors), whichever is smaller. The first function created maximizes the differences between groups on that function. The second function maximizes differences on that function, but also must not be correlated with the previous function. This continues with subsequent functions with the requirement that the new function not be correlated with any of the previous functions.
Given group
, with
sets of sample space, there is a discriminant rule such that if
, then
. Discriminant analysis then, finds “good” regions of
to minimize classification error, therefore leading to a high percent correct classified in the classification table.
Each function is given a discriminant score to determine how well it predicts group placement.
*Structure Correlation Coefficients: The correlation between each predictor and the discriminant score of each function. This is a zero-order correlation (i.e., not corrected for the other predictors).
*Standardized Coefficients: Each predictor's weight in the linear combination that is the discriminant function. Like in a regression equation, these coefficients are partial (i.e., corrected for the other predictors). Indicates the unique contribution of each predictor in predicting group assignment.
*Functions at Group Centroids: Mean discriminant scores for each grouping variable are given for each function. The farther apart the means are, the less error there will be in classification.
Discrimination rules
*
Maximum likelihood
In statistics, maximum likelihood estimation (MLE) is a method of estimation theory, estimating the Statistical parameter, parameters of an assumed probability distribution, given some observed data. This is achieved by Mathematical optimization, ...
: Assigns
to the group that maximizes population (group) density.
[Hardle, W., Simar, L. (2007). ]
Applied Multivariate Statistical Analysis
'. Springer Berlin Heidelberg. pp. 289-303.
*Bayes Discriminant Rule: Assigns
to the group that maximizes
, where ''π
i'' represents the
prior probability
In Bayesian statistical inference, a prior probability distribution, often simply called the prior, of an uncertain quantity is the probability distribution that would express one's beliefs about this quantity before some evidence is taken into ...
of that classification, and
represents the population density.
*
Fisher's linear discriminant rule: Maximizes the ratio between ''SS''
between and ''SS''
within, and finds a linear combination of the predictors to predict group.
Eigenvalues
An
eigenvalue
In linear algebra, an eigenvector () or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted b ...
in discriminant analysis is the characteristic root of each function. It is an indication of how well that function differentiates the groups, where the larger the eigenvalue, the better the function differentiates.
This however, should be interpreted with caution, as eigenvalues have no upper limit.
The eigenvalue can be viewed as a ratio of ''SS''
between and ''SS''
within as in ANOVA when the dependent variable is the discriminant function, and the groups are the levels of the
IV.
This means that the largest eigenvalue is associated with the first function, the second largest with the second, etc..
Effect size
Some suggest the use of eigenvalues as
effect size
In statistics, an effect size is a value measuring the strength of the relationship between two variables in a population, or a sample-based estimate of that quantity. It can refer to the value of a statistic calculated from a sample of data, the ...
measures, however, this is generally not supported.
Instead, the
canonical correlation
In statistics, canonical-correlation analysis (CCA), also called canonical variates analysis, is a way of inferring information from cross-covariance matrices. If we have two vectors ''X'' = (''X''1, ..., ''X'n'') and ''Y' ...
is the preferred measure of effect size. It is similar to the eigenvalue, but is the square root of the ratio of ''SS''
between and ''SS''
total. It is the correlation between groups and the function.
Another popular measure of effect size is the percent of variance for each function. This is calculated by: (''λ
x/Σλ
i'') X 100 where ''λ
x'' is the eigenvalue for the function and Σ''λ
i'' is the sum of all eigenvalues. This tells us how strong the prediction is for that particular function compared to the others.
Percent correctly classified can also be analyzed as an effect size. The kappa value can describe this while correcting for chance agreement.
Canonical discriminant analysis for ''k'' classes
Canonical discriminant analysis (CDA) finds axes (''k'' − 1
canonical coordinates, ''k'' being the number of classes) that best separate the categories. These linear functions are uncorrelated and define, in effect, an optimal ''k'' − 1 space through the ''n''-dimensional cloud of data that best separates (the projections in that space of) the ''k'' groups. See “
Multiclass LDA” for details below.
Fisher's linear discriminant
The terms ''Fisher's linear discriminant'' and ''LDA'' are often used interchangeably, although
Fisher's original article
actually describes a slightly different discriminant, which does not make some of the assumptions of LDA such as
normally distributed classes or equal class
covariance
In probability theory and statistics, covariance is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the ...
s.
Suppose two classes of observations have
mean
There are several kinds of mean in mathematics, especially in statistics. Each mean serves to summarize a given group of data, often to better understand the overall value (magnitude and sign) of a given data set.
For a data set, the ''arithme ...
s
and covariances
. Then the linear combination of features
will have
mean
There are several kinds of mean in mathematics, especially in statistics. Each mean serves to summarize a given group of data, often to better understand the overall value (magnitude and sign) of a given data set.
For a data set, the ''arithme ...
s
and
variance
In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers ...
s
for
. Fisher defined the separation between these two
distributions to be the ratio of the variance between the classes to the variance within the classes:
:
This measure is, in some sense, a measure of the
signal-to-noise ratio
Signal-to-noise ratio (SNR or S/N) is a measure used in science and engineering that compares the level of a desired signal to the level of background noise. SNR is defined as the ratio of signal power to the noise power, often expressed in deci ...
for the class labelling. It can be shown that the maximum separation occurs when
:
When the assumptions of LDA are satisfied, the above equation is equivalent to LDA.
![Fisher2classes](https://upload.wikimedia.org/wikipedia/commons/f/f4/Fisher2classes.png)
Be sure to note that the vector
is the
normal Normal(s) or The Normal(s) may refer to:
Film and television
* ''Normal'' (2003 film), starring Jessica Lange and Tom Wilkinson
* ''Normal'' (2007 film), starring Carrie-Anne Moss, Kevin Zegers, Callum Keith Rennie, and Andrew Airlie
* ''Norma ...
to the discriminant
hyperplane
In geometry, a hyperplane is a subspace whose dimension is one less than that of its ''ambient space''. For example, if a space is 3-dimensional then its hyperplanes are the 2-dimensional planes, while if the space is 2-dimensional, its hyper ...
. As an example, in a two dimensional problem, the line that best divides the two groups is perpendicular to
.
Generally, the data points to be discriminated are projected onto
; then the threshold that best separates the data is chosen from analysis of the one-dimensional distribution. There is no general rule for the threshold. However, if projections of points from both classes exhibit approximately the same distributions, a good choice would be the hyperplane between projections of the two means,
and
. In this case the parameter c in threshold condition
can be found explicitly:
:
.
Otsu's method
In computer vision and image processing, Otsu's method, named after , is used to perform automatic image thresholding. In the simplest form, the algorithm returns a single intensity threshold that separate pixels into two classes, foreground a ...
is related to Fisher's linear discriminant, and was created to binarize the histogram of pixels in a grayscale image by optimally picking the black/white threshold that minimizes intra-class variance and maximizes inter-class variance within/between grayscales assigned to black and white pixel classes.
Multiclass LDA
![3dProjections](https://upload.wikimedia.org/wikipedia/commons/4/4c/3dProjections.png)
In the case where there are more than two classes, the analysis used in the derivation of the Fisher discriminant can be extended to find a
subspace which appears to contain all of the class variability.
[Garson, G. D. (2008). Discriminant function analysis. .] This generalization is due to
C. R. Rao
Calyampudi Radhakrishna Rao FRS (born 10 September 1920), commonly known as C. R. Rao, is an Indian-American mathematician and statistician. He is currently professor emeritus at Pennsylvania State University and Research Professor at the Un ...
.
Suppose that each of C classes has a mean
and the same covariance
. Then the scatter between class variability may be defined by the sample covariance of the class means
:
where
is the mean of the class means. The class separation in a direction
in this case will be given by
:
This means that when
is an
eigenvector
In linear algebra, an eigenvector () or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted b ...
of
the separation will be equal to the corresponding
eigenvalue
In linear algebra, an eigenvector () or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted b ...
.
If
is diagonalizable, the variability between features will be contained in the subspace spanned by the eigenvectors corresponding to the ''C'' − 1 largest eigenvalues (since
is of rank ''C'' − 1 at most). These eigenvectors are primarily used in feature reduction, as in PCA. The eigenvectors corresponding to the smaller eigenvalues will tend to be very sensitive to the exact choice of training data, and it is often necessary to use regularisation as described in the next section.
If classification is required, instead of
dimension reduction
Dimensionality reduction, or dimension reduction, is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally ...
, there are a number of alternative techniques available. For instance, the classes may be partitioned, and a standard Fisher discriminant or LDA used to classify each partition. A common example of this is "one against the rest" where the points from one class are put in one group, and everything else in the other, and then LDA applied. This will result in C classifiers, whose results are combined. Another common
method is pairwise classification, where a new classifier is created for each pair of classes (giving ''C''(''C'' − 1)/2 classifiers in total), with the individual classifiers combined to produce a final classification.
Incremental LDA
The typical implementation of the LDA technique requires that all the samples are available in advance. However, there are situations where the entire data set is not available and the input data are observed as a stream. In this case, it is desirable for the LDA feature extraction to have the ability to update the computed LDA features by observing the new samples without running the algorithm on the whole data set. For example, in many real-time applications such as mobile robotics or on-line face recognition, it is important to update the extracted LDA features as soon as new observations are available. An LDA feature extraction technique that can update the LDA features by simply observing new samples is an ''incremental LDA algorithm'', and this idea has been extensively studied over the last two decades.
Chatterjee and Roychowdhury proposed an incremental self-organized LDA algorithm for updating the LDA features.
In other work, Demir and Ozmehmet proposed online local learning algorithms for updating LDA features incrementally using error-correcting and the Hebbian learning rules. Later, Aliyari ''et a''l. derived fast incremental algorithms to update the LDA features by observing the new samples.
Practical use
In practice, the class means and covariances are not known. They can, however, be estimated from the training set. Either the
maximum likelihood estimate
In statistics, maximum likelihood estimation (MLE) is a method of estimating the parameters of an assumed probability distribution, given some observed data. This is achieved by maximizing a likelihood function so that, under the assumed statist ...
or the
maximum a posteriori
In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is an estimate of an unknown quantity, that equals the mode of the posterior distribution. The MAP can be used to obtain a point estimate of an unobserved quantity on the ...
estimate may be used in place of the exact value in the above equations. Although the estimates of the covariance may be considered optimal in some sense, this does not mean that the resulting discriminant obtained by substituting these values is optimal in any sense, even if the assumption of normally distributed classes is correct.
Another complication in applying LDA and Fisher's discriminant to real data occurs when the number of measurements of each sample (i.e., the dimensionality of each data vector) exceeds the number of samples in each class.
In this case, the covariance estimates do not have full rank, and so cannot be inverted. There are a number of ways to deal with this. One is to use a
pseudo inverse
In mathematics, and in particular, algebra, a generalized inverse (or, g-inverse) of an element ''x'' is an element ''y'' that has some properties of an inverse element but not necessarily all of them. The purpose of constructing a generalized in ...
instead of the usual matrix inverse in the above formulae. However, better numeric stability may be achieved by first projecting the problem onto the subspace spanned by
.
Another strategy to deal with small sample size is to use a
shrinkage estimator of the covariance matrix, which
can be expressed mathematically as
:
where
is the identity matrix, and
is the ''shrinkage intensity'' or ''regularisation parameter''.
This leads to the framework of regularized discriminant analysis
or shrinkage discriminant analysis.
Also, in many practical cases linear discriminants are not suitable. LDA and Fisher's discriminant can be extended for use in non-linear classification via the
kernel trick
In machine learning, kernel machines are a class of algorithms for pattern analysis, whose best known member is the support-vector machine (SVM). The general task of pattern analysis is to find and study general types of relations (for example ...
. Here, the original observations are effectively mapped into a higher dimensional non-linear space. Linear classification in this non-linear space is then equivalent to non-linear classification in the original space. The most commonly used example of this is the
kernel Fisher discriminant.
LDA can be generalized to
multiple discriminant analysis
Multiple Discriminant Analysis (MDA) is a multivariate dimensionality reduction technique. It has been used to predict signals as diverse as neural memory traces and corporate failure.
MDA is not directly used to perform classification. It merel ...
, where ''c'' becomes a
categorical variable
In statistics, a categorical variable (also called qualitative variable) is a variable that can take on one of a limited, and usually fixed, number of possible values, assigning each individual or other unit of observation to a particular group or ...
with ''N'' possible states, instead of only two. Analogously, if the class-conditional densities
are normal with shared covariances, the
sufficient statistic
In statistics, a statistic is ''sufficient'' with respect to a statistical model and its associated unknown parameter if "no other statistic that can be calculated from the same sample provides any additional information as to the value of the pa ...
for
are the values of ''N'' projections, which are the
subspace spanned by the ''N'' means,
affine projected by the inverse covariance matrix. These projections can be found by solving a
generalized eigenvalue problem
In linear algebra, eigendecomposition is the factorization of a matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors. Only diagonalizable matrices can be factorized in this way. When the mat ...
, where the numerator is the covariance matrix formed by treating the means as the samples, and the denominator is the shared covariance matrix. See “
Multiclass LDA” above for details.
Applications
In addition to the examples given below, LDA is applied in
positioning and
product management
Product management is the business process of planning, developing, launching, and managing a product or service. It includes the entire lifecycle of a product, from ideation to development to go to market. Product managers are responsible for ...
.
Bankruptcy prediction
In
bankruptcy prediction
Bankruptcy prediction is the art of predicting bankruptcy and various measures of financial distress of public firms. It is a vast area of finance and accounting research. The importance of the area is due in part to the relevance for creditors ...
based on accounting ratios and other financial variables, linear discriminant analysis was the first statistical method applied to systematically explain which firms entered bankruptcy vs. survived. Despite limitations including known nonconformance of accounting ratios to the normal distribution assumptions of LDA,
Edward Altman
Edward I. Altman (born June 5, 1941) is a Professor of Finance, Emeritus, at New York University's Stern School of Business. He is best known for the development of the Altman Z-score for predicting bankruptcy which he published in 1968. Profe ...
's
1968 model is still a leading model in practical applications.
Face recognition
In computerised
face recognition
A facial recognition system is a technology capable of matching a human face from a digital image or a video frame against a database of faces. Such a system is typically employed to authenticate users through ID verification services, and wo ...
, each face is represented by a large number of pixel values. Linear discriminant analysis is primarily used here to reduce the number of features to a more manageable number before classification. Each of the new dimensions is a linear combination of pixel values, which form a template. The linear combinations obtained using Fisher's linear discriminant are called ''Fisher faces'', while those obtained using the related
principal component analysis
Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and ...
are called ''
eigenfaces
An eigenface () is the name given to a set of eigenvectors when used in the computer vision problem of human face recognition. The approach of using eigenfaces for recognition was developed by Sirovich and Kirby and used by Matthew Turk and A ...
''.
Marketing
In
marketing
Marketing is the process of exploring, creating, and delivering value to meet the needs of a target market in terms of goods and services; potentially including selection of a target audience; selection of certain attributes or themes to emph ...
, discriminant analysis was once often used to determine the factors which distinguish different types of customers and/or products on the basis of surveys or other forms of collected data.
Logistic regression
In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear function (calculus), linear combination of one or more independent var ...
or other methods are now more commonly used. The use of discriminant analysis in marketing can be described by the following steps:
#Formulate the problem and gather data—Identify the
salient attributes consumers use to evaluate products in this category—Use
quantitative marketing research
Quantitative marketing research is the application of quantitative research techniques to the field of marketing research. It has roots in both the positivist view of the world, and the modern marketing viewpoint that marketing is an interactive ...
techniques (such as
surveys) to collect data from a sample of potential customers concerning their ratings of all the product attributes. The data collection stage is usually done by marketing research professionals. Survey questions ask the respondent to rate a product from one to five (or 1 to 7, or 1 to 10) on a range of attributes chosen by the researcher. Anywhere from five to twenty attributes are chosen. They could include things like: ease of use, weight, accuracy, durability, colourfulness, price, or size. The attributes chosen will vary depending on the product being studied. The same question is asked about all the products in the study. The data for multiple products is codified and input into a statistical program such as
R,
SPSS
SPSS Statistics is a statistical software suite developed by IBM for data management, advanced analytics, multivariate analysis, business intelligence, and criminal investigation. Long produced by SPSS Inc., it was acquired by IBM in 2009. C ...
or
SAS. (This step is the same as in Factor analysis).
#Estimate the Discriminant Function Coefficients and determine the statistical significance and validity—Choose the appropriate discriminant analysis method. The direct method involves estimating the discriminant function so that all the predictors are assessed simultaneously. The
stepwise method enters the predictors sequentially. The two-group method should be used when the dependent variable has two categories or states. The multiple discriminant method is used when the dependent variable has three or more categorical states. Use
Wilks's Lambda to test for significance in SPSS or F stat in SAS. The most common method used to test validity is to split the sample into an estimation or analysis sample, and a validation or holdout sample. The estimation sample is used in constructing the discriminant function. The validation sample is used to construct a classification matrix which contains the number of correctly classified and incorrectly classified cases. The percentage of correctly classified cases is called the ''hit ratio''.
#Plot the results on a two dimensional map, define the dimensions, and interpret the results. The statistical program (or a related module) will map the results. The map will plot each product (usually in two-dimensional space). The distance of products to each other indicate either how different they are. The dimensions must be labelled by the researcher. This requires subjective judgement and is often very challenging. See
perceptual mapping.
Biomedical studies
The main application of discriminant analysis in medicine is the assessment of severity state of a patient and prognosis of disease outcome. For example, during retrospective analysis, patients are divided into groups according to severity of disease – mild, moderate and severe form. Then results of clinical and laboratory analyses are studied in order to reveal variables which are statistically different in studied groups. Using these variables, discriminant functions are built which help to objectively classify disease in a future patient into mild, moderate or severe form.
In biology, similar principles are used in order to classify and define groups of different biological objects, for example, to define phage types of Salmonella enteritidis based on Fourier transform infrared spectra, to detect animal source of ''Escherichia coli'' studying its virulence factors etc.
Earth science
This method can be used to . For example, when different data from various zones are available, discriminant analysis can find the pattern within the data and classify it effectively.
Comparison to logistic regression
Discriminant function analysis is very similar to
logistic regression
In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear function (calculus), linear combination of one or more independent var ...
, and both can be used to answer the same research questions.
Logistic regression does not have as many assumptions and restrictions as discriminant analysis. However, when discriminant analysis’ assumptions are met, it is more powerful than logistic regression. Unlike logistic regression, discriminant analysis can be used with small sample sizes. It has been shown that when sample sizes are equal, and homogeneity of variance/covariance holds, discriminant analysis is more accurate.
Despite all these advantages, logistic regression has none-the-less become the common choice, since the assumptions of discriminant analysis are rarely met.
Linear discriminant in high dimension
Geometric anomalies in higher dimensions lead to the well-known
curse of dimensionality
The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience. The ...
. Nevertheless, proper utilization of
concentration of measure
In mathematics, concentration of measure (about a median) is a principle that is applied in measure theory, probability and combinatorics, and has consequences for other fields such as Banach space theory. Informally, it states that "A random v ...
phenomena can make computation easier. An important case of these
''blessing of dimensionality'' phenomena was highlighted by Donoho and Tanner: if a sample is essentially high-dimensional then each point can be separated from the rest of the sample by linear inequality, with high probability, even for exponentially large samples. These linear inequalities can be selected in the standard (Fisher's) form of the linear discriminant for a rich family of probability distribution. In particular, such theorems are proven for
log-concave distributions including
multidimensional normal distribution (the proof is based on the concentration inequalities for log-concave measures
[Guédon, O., Milman, E. (2011]
Interpolating thin-shell and sharp large-deviation estimates for isotropic log-concave measures
Geom. Funct. Anal. 21 (5), 1043–1068.) and for product measures on a multidimensional cube (this is proven using
Talagrand's concentration inequality
In the probability theory field of mathematics , Talagrand's concentration inequality is an isoperimetric-type inequality (mathematics), inequality for product space, product probability spaces. It was first proved by the French mathematician Miche ...
for product probability spaces). Data separability by classical linear discriminants simplifies the problem of error correction for
artificial intelligence
Artificial intelligence (AI) is intelligence—perceiving, synthesizing, and inferring information—demonstrated by machines, as opposed to intelligence displayed by animals and humans. Example tasks in which this is done include speech re ...
systems in high dimension.
See also
*
Data mining
*
Decision tree learning
Decision tree learning is a supervised learning approach used in statistics, data mining and machine learning. In this formalism, a classification or regression decision tree is used as a predictive model to draw conclusions about a set of ob ...
*
Factor analysis
Factor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved variables called factors. For example, it is possible that variations in six observed ...
*
Kernel Fisher discriminant analysis In statistics, kernel Fisher discriminant analysis (KFD), also known as generalized discriminant analysis and kernel discriminant analysis, is a kernelized version of linear discriminant analysis (LDA). It is named after Ronald Fisher.
Linear di ...
*
Logit
In statistics, the logit ( ) function is the quantile function associated with the standard logistic distribution. It has many uses in data analysis and machine learning, especially in data transformations.
Mathematically, the logit is the ...
(for
logistic regression
In statistics, the logistic model (or logit model) is a statistical model that models the probability of an event taking place by having the log-odds for the event be a linear function (calculus), linear combination of one or more independent var ...
)
*
Linear regression
In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is call ...
*
Multiple discriminant analysis
Multiple Discriminant Analysis (MDA) is a multivariate dimensionality reduction technique. It has been used to predict signals as diverse as neural memory traces and corporate failure.
MDA is not directly used to perform classification. It merel ...
*
Multidimensional scaling
Multidimensional scaling (MDS) is a means of visualizing the level of similarity of individual cases of a dataset. MDS is used to translate "information about the pairwise 'distances' among a set of n objects or individuals" into a configurati ...
*
Pattern recognition
Pattern recognition is the automated recognition of patterns and regularities in data. It has applications in statistical data analysis, signal processing, image analysis, information retrieval, bioinformatics, data compression, computer graphi ...
*
Preference regression
Preference regression is a statistical technique used by marketers to determine consumers’ preferred core benefits. It usually supplements product positioning techniques like multi dimensional scaling or factor analysis and is used to create ...
*
Quadratic classifier
In statistics, a quadratic classifier is a statistical classifier that uses a quadratic decision surface to separate measurements of two or more classes of objects or events. It is a more general version of the linear classifier.
The classific ...
*
Statistical classification
In statistics, classification is the problem of identifying which of a set of categories (sub-populations) an observation (or observations) belongs to. Examples are assigning a given email to the "spam" or "non-spam" class, and assigning a diagno ...
References
Further reading
*
*
*
*
*
*
External links
Discriminant Correlation Analysis (DCA) of the Haghighat article (see above)ALGLIBcontains open-source LDA implementation in C# / C++ / Pascal / VBA.
LDA in Python LDA implementation in Python
*
*
ttp://people.revoledu.com/kardi/tutorial/LDA/ Discriminant analysis tutorial in Microsoft Excel by Kardi TeknomoCourse notes, Discriminant function analysis by David W. Stockburger, Missouri State UniversityDiscriminant function analysis (DA) by John Poulsen and Aaron French, San Francisco State University
{{DEFAULTSORT:Linear Discriminant Analysis
Classification algorithms
Market research
Market segmentation
Statistical classification