An eigenface () is the name given to a set of
eigenvectors when used in the
computer vision
Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the human ...
problem of human
face recognition
A facial recognition system is a technology capable of matching a human face from a digital image or a video frame against a database of faces. Such a system is typically employed to authenticate users through ID verification services, an ...
. The approach of using eigenfaces for
recognition
Recognition may refer to:
*Award, something given in recognition of an achievement
Machine learning
*Pattern recognition, a branch of machine learning which encompasses the meanings below
Biometric
* Recognition of human individuals, or biomet ...
was developed by Sirovich and Kirby and used by
Matthew Turk and
Alex Pentland
Alex Paul "Sandy" Pentland (born 1951) is an American computer scientist, the Toshiba Professor at MIT, and serial entrepreneur.
Education
Pentland received his bachelor's degree from the University of Michigan and obtained his Ph.D. from ...
in face classification.
The eigenvectors are derived from the
covariance matrix
In probability theory and statistics, a covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix, or variance–covariance matrix) is a square matrix giving the covariance between each pair of elements of ...
of the
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
over the high-
dimension
In physics and mathematics, the dimension of a mathematical space (or object) is informally defined as the minimum number of coordinates needed to specify any point within it. Thus, a line has a dimension of one (1D) because only one coord ...
al
vector space
In mathematics and physics, a vector space (also called a linear space) is a set whose elements, often called '' vectors'', may be added together and multiplied ("scaled") by numbers called ''scalars''. Scalars are often real numbers, but can ...
of face images. The eigenfaces themselves form a basis set of all images used to construct the covariance matrix. This produces dimension reduction by allowing the smaller set of basis images to represent the original training images. Classification can be achieved by comparing how faces are represented by the basis set.
History
The eigenface approach began with a search for a low-dimensional representation of face images. Sirovich and Kirby showed that
principal component analysis
Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and ...
could be used on a collection of face images to form a set of basis features.
These basis images, known as eigenpictures, could be linearly combined to reconstruct images in the original training set. If the training set consists of ''M'' images, principal component analysis could form a basis set of ''N'' images, where ''N < M''. The reconstruction error is reduced by increasing the number of eigenpictures; however, the number needed is always chosen less than ''M''. For example, if you need to generate a number of ''N'' eigenfaces for a training set of ''M'' face images, you can say that each face image can be made up of "proportions" of all the ''K'' "features" or eigenfaces: Face image
1 = (23% of E
1) + (2% of E
2) + (51% of E
3) + ... + (1% E
n).
In 1991 M. Turk and A. Pentland expanded these results and presented the eigenface method of face recognition.
In addition to designing a system for automated face recognition using eigenfaces, they showed a way of calculating the
eigenvectors of a
covariance matrix
In probability theory and statistics, a covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix, or variance–covariance matrix) is a square matrix giving the covariance between each pair of elements of ...
such that computers of the time could perform eigen-decomposition on a large number of face images. Face images usually occupy a high-dimensional space and conventional principal component analysis was intractable on such data sets. Turk and Pentland's paper demonstrated ways to extract the eigenvectors based on matrices sized by the number of images rather than the number of pixels.
Once established, the eigenface method was expanded to include methods of preprocessing to improve accuracy. Multiple manifold approaches were also used to build sets of eigenfaces for different subjects and different features, such as the eyes.
Generation
A set of eigenfaces can be generated by performing a mathematical process called
principal component analysis
Principal component analysis (PCA) is a popular technique for analyzing large datasets containing a high number of dimensions/features per observation, increasing the interpretability of data while preserving the maximum amount of information, and ...
(PCA) on a large set of images depicting different human faces. Informally, eigenfaces can be considered a set of "standardized face ingredients", derived from
statistical analysis
Statistical inference is the process of using data analysis to infer properties of an underlying distribution of probability.Upton, G., Cook, I. (2008) ''Oxford Dictionary of Statistics'', OUP. . Inferential statistical analysis infers propertie ...
of many pictures of faces. Any human face can be considered to be a combination of these standard faces. For example, one's face might be composed of the average face plus 10% from eigenface 1, 55% from eigenface 2, and even −3% from eigenface 3. Remarkably, it does not take many eigenfaces combined together to achieve a fair approximation of most faces. Also, because a person's face is not recorded by a
digital photograph
Digital photography uses cameras containing arrays of electronic photodetectors interfaced to an analog-to-digital converter (ADC) to produce images focused by a lens, as opposed to an exposure on photographic film. The digitized image is stor ...
, but instead as just a list of values (one value for each eigenface in the database used), much less space is taken for each person's face.
The eigenfaces that are created will appear as light and dark areas that are arranged in a specific pattern. This pattern is how different features of a face are singled out to be evaluated and scored. There will be a pattern to evaluate
symmetry
Symmetry (from grc, συμμετρία "agreement in dimensions, due proportion, arrangement") in everyday language refers to a sense of harmonious and beautiful proportion and balance. In mathematics, "symmetry" has a more precise definiti ...
, whether there is any style of facial hair, where the hairline is, or an evaluation of the size of the nose or mouth. Other eigenfaces have patterns that are less simple to identify, and the image of the eigenface may look very little like a face.
The technique used in creating eigenfaces and using them for recognition is also used outside of face recognition:
handwriting recognition,
lip reading
The lips are the visible body part at the mouth of many animals, including humans. Lips are soft, movable, and serve as the opening for food intake and in the articulation of sound and speech. Human lips are a tactile sensory organ, and can be ...
,
voice recognition,
sign language
Sign languages (also known as signed languages) are languages that use the visual-manual modality to convey meaning, instead of spoken words. Sign languages are expressed through manual articulation in combination with non-manual markers. Sign ...
/hand
gestures
A gesture is a form of non-verbal communication or non-vocal communication in which visible bodily actions communicate particular messages, either in place of, or in conjunction with, speech. Gestures include movement of the hands, face, o ...
interpretation and
medical imaging
Medical imaging is the technique and process of imaging the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology). Medical imaging seeks to re ...
analysis. Therefore, some do not use the term eigenface, but prefer to use 'eigenimage'.
Practical implementation
To create a set of eigenfaces, one must:
# Prepare a training set of face images. The pictures constituting the training set should have been taken under the same lighting conditions, and must be normalized to have the eyes and mouths aligned across all images. They must also be all resampled to a common
pixel
In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a raster image, or the smallest point in an all points addressable display device.
In most digital display devices, pixels are the ...
resolution (''r'' × ''c''). Each image is treated as one vector, simply by
concatenating the rows of pixels in the original image, resulting in a single column with ''r'' × ''c'' elements. For this implementation, it is assumed that all images of the training set are stored in a single
matrix
Matrix most commonly refers to:
* ''The Matrix'' (franchise), an American media franchise
** '' The Matrix'', a 1999 science-fiction action film
** "The Matrix", a fictional setting, a virtual reality environment, within ''The Matrix'' (franchi ...
T, where each column of the matrix is an image.
# Subtract the
mean
There are several kinds of mean in mathematics, especially in statistics. Each mean serves to summarize a given group of data, often to better understand the overall value ( magnitude and sign) of a given data set.
For a data set, the '' ar ...
. The average image a has to be calculated and then subtracted from each original image in T.
# Calculate the
eigenvectors and eigenvalues of the
covariance matrix
In probability theory and statistics, a covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix, or variance–covariance matrix) is a square matrix giving the covariance between each pair of elements of ...
S. Each eigenvector has the same dimensionality (number of components) as the original images, and thus can itself be seen as an image. The eigenvectors of this covariance matrix are therefore called eigenfaces. They are the directions in which the images differ from the mean image. Usually this will be a computationally expensive step (if at all possible), but the practical applicability of eigenfaces stems from the possibility to compute the eigenvectors of S efficiently, without ever computing S explicitly, as detailed below.
# Choose the principal components. Sort the eigenvalues in descending order and arrange eigenvectors accordingly. The number of principal components ''k'' is determined arbitrarily by setting a threshold ε on the total variance. Total variance , = number of components.
# k is the smallest number that satisfies
These eigenfaces can now be used to represent both existing and new faces: we can project a new (mean-subtracted) image on the eigenfaces and thereby record how that new face differs from the mean face. The eigenvalues associated with each eigenface represent how much the images in the training set vary from the mean image in that direction. Information is lost by projecting the image on a subset of the eigenvectors, but losses are minimized by keeping those eigenfaces with the largest eigenvalues. For instance, working with a 100 × 100 image will produce 10,000 eigenvectors. In practical applications, most faces can typically be identified using a projection on between 100 and 150 eigenfaces, so that most of the 10,000 eigenvectors can be discarded.
Matlab example code
Here is an example of calculating eigenfaces with Extended Yale Face Database B. To evade computational and storage bottleneck, the face images are sampled down by a factor 4×4=16.
clear all;
close all;
load yalefaces
, w, n= size(yalefaces);
d = h * w;
% vectorize images
x = reshape(yalefaces, n;
x = double(x);
% subtract mean
mean_matrix = mean(x, 2);
x = bsxfun(@minus, x, mean_matrix);
% calculate covariance
s = cov(x');
% obtain eigenvalue & eigenvector
, D
The comma is a punctuation mark that appears in several variants in different languages. It has the same shape as an apostrophe or single closing quotation mark () in many typefaces, but it differs from them in being placed on the baseline o ...
= eig(s);
eigval = diag(D);
% sort eigenvalues in descending order
eigval = eigval(end: - 1:1);
V = fliplr(V);
% show mean and 1st through 15th principal eigenvectors
figure, subplot(4, 4, 1)
imagesc(reshape(mean_matrix, , w
The comma is a punctuation mark that appears in several variants in different languages. It has the same shape as an apostrophe or single closing quotation mark () in many typefaces, but it differs from them in being placed on the baseline o ...
)
colormap gray
for i = 1:15
subplot(4, 4, i + 1)
imagesc(reshape(V(:, i), h, w))
end
Note that although the covariance matrix S generates many eigenfaces, only a fraction of those are needed to represent the majority of the faces. For example, to represent 95% of the total variation of all face images, only the first 43 eigenfaces are needed. To calculate this result, implement the following code:
% evaluate the number of principal components needed to represent 95% Total variance.
eigsum = sum(eigval);
csum = 0;
for i = 1:d
csum = csum + eigval(i);
tv = csum / eigsum;
if tv > 0.95
k95 = i;
break
end;
end;
Computing the eigenvectors
Performing PCA directly on the covariance matrix of the images is often computationally infeasible. If small images are used, say 100 × 100 pixels, each image is a point in a 10,000-dimensional space and the covariance matrix S is a matrix of 10,000 × 10,000 = 10
8 elements. However the
rank of the covariance matrix is limited by the number of training examples: if there are ''N'' training examples, there will be at most ''N'' − 1 eigenvectors with non-zero eigenvalues. If the number of training examples is smaller than the dimensionality of the images, the principal components can be computed more easily as follows.
Let T be the matrix of preprocessed training examples, where each column contains one mean-subtracted image. The covariance matrix can then be computed as S = TT
T and the eigenvector decomposition of S is given by
:
However TT
T is a large matrix, and if instead we take the eigenvalue decomposition of
:
then we notice that by pre-multiplying both sides of the equation with T, we obtain
:
Meaning that, if u
i is an eigenvector of T
TT, then v
i = Tu
i is an eigenvector of S. If we have a training set of 300 images of 100 × 100 pixels, the matrix T
TT is a 300 × 300 matrix, which is much more manageable than the 10,000 × 10,000 covariance matrix. Notice however that the resulting vectors v
i are not normalised; if normalisation is required it should be applied as an extra step.
Connection with SVD
Let denote the data matrix with column as the image vector with mean subtracted. Then,
:
Let the
singular value decomposition
In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix. It generalizes the eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any \ m \times n\ matrix. It is re ...
(SVD) of be:
:
Then the eigenvalue decomposition for
is:
:
, where Λ=diag (eigenvalues of
)
Thus we can see easily that:
: The eigenfaces = the first
(
) columns of
associated with the nonzero singular values.
: The ith eigenvalue of
ith singular value of
Using SVD on data matrix , it is unnecessary to calculate the actual covariance matrix to get eigenfaces.
Use in facial recognition
Facial recognition was the motivation for the creation of eigenfaces. For this use, eigenfaces have advantages over other techniques available, such as the system's speed and efficiency. As eigenface is primarily a dimension reduction method, a system can represent many subjects with a relatively small set of data. As a face-recognition system it is also fairly invariant to large reductions in image sizing; however, it begins to fail considerably when the variation between the seen images and probe image is large.
To recognise faces, gallery images – those seen by the system – are saved as collections of weights describing the contribution each eigenface has to that image. When a new face is presented to the system for classification, its own weights are found by projecting the image onto the collection of eigenfaces. This provides a set of weights describing the probe face. These weights are then classified against all weights in the gallery set to find the closest match. A nearest-neighbour method is a simple approach for finding the
Euclidean distance
In mathematics, the Euclidean distance between two points in Euclidean space is the length of a line segment between the two points.
It can be calculated from the Cartesian coordinates of the points using the Pythagorean theorem, therefore ...
between two vectors, where the minimum can be classified as the closest subject.
Intuitively, the recognition process with the eigenface method is to project query images into the face-space spanned by eigenfaces calculated, and to find the closest match to a face class in that face-space.
;Pseudo code
:* Given input image vector
, the mean image vector from the database
, calculate the weight of the kth eigenface as:
:*:
:*:Then form a weight vector