In
statistics, the Wishart distribution is a generalization to multiple dimensions of the
gamma distribution. It is named in honor of
John Wishart, who first formulated the distribution in 1928.
It is a family of
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
s defined over symmetric,
nonnegative-definite random matrices
In probability theory and mathematical physics, a random matrix is a matrix-valued random variable—that is, a matrix in which some or all elements are random variables. Many important properties of physical systems can be represented mathemat ...
(i.e.
matrix-valued
random variables). In random matrix theory, the space of Wishart matrices is called the ''Wishart ensemble''.
These distributions are of great importance in the
estimation of covariance matrices in
multivariate statistics. In
Bayesian statistics
Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
, the Wishart distribution is the
conjugate prior
In Bayesian probability theory, if the posterior distribution p(\theta \mid x) is in the same probability distribution family as the prior probability distribution p(\theta), the prior and posterior are then called conjugate distributions, and ...
of the
inverse
Inverse or invert may refer to:
Science and mathematics
* Inverse (logic), a type of conditional sentence which is an immediate inference made from another conditional sentence
* Additive inverse (negation), the inverse of a number that, when a ...
covariance-matrix of a
multivariate-normal random-vector.
Definition
Suppose is a matrix, each column of which is
independently drawn from a
-variate normal distribution with zero mean:
:
Then the Wishart distribution is the
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
of the random matrix
:
known as the
scatter matrix. One indicates that has that probability distribution by writing
:
The positive integer is the number of ''
degrees of freedom''. Sometimes this is written . For the matrix is invertible with probability if is invertible.
If then this distribution is a
chi-squared distribution
In probability theory and statistics, the chi-squared distribution (also chi-square or \chi^2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables. The chi-square ...
with degrees of freedom.
Occurrence
The Wishart distribution arises as the distribution of the sample covariance matrix for a sample from a
multivariate normal distribution. It occurs frequently in
likelihood-ratio tests in multivariate statistical analysis. It also arises in the spectral theory of
random matrices
In probability theory and mathematical physics, a random matrix is a matrix-valued random variable—that is, a matrix in which some or all elements are random variables. Many important properties of physical systems can be represented mathemat ...
and in multidimensional Bayesian analysis. It is also encountered in wireless communications, while analyzing the performance of
Rayleigh fading MIMO wireless channels .
Probability density function
The Wishart distribution can be
characterized by its
probability density function as follows:
Let be a symmetric matrix of random variables that is
positive semi-definite. Let be a (fixed) symmetric positive definite matrix of size .
Then, if , has a Wishart distribution with degrees of freedom if it has the
probability density function
:
where
is the
determinant of
and is the
multivariate gamma function defined as
:
The density above is not the joint density of all the
elements of the random matrix (such density does not exist because of the symmetry constrains
), it is rather the joint density of
elements
for
(,
page 38). Also, the density formula above applies only to positive definite matrices
for other matrices the density is equal to zero.
The joint-eigenvalue density for the eigenvalues
of a random matrix
is,
:
where
is a constant.
In fact the above definition can be extended to any real . If , then the Wishart no longer has a density—instead it represents a singular distribution that takes values in a lower-dimension subspace of the space of matrices.
Use in Bayesian statistics
In
Bayesian statistics
Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
, in the context of the
multivariate normal distribution, the Wishart distribution is the conjugate prior to the precision matrix , where is the covariance matrix.
Choice of parameters
The least informative, proper Wishart prior is obtained by setting .
The prior mean of is , suggesting that a reasonable choice for would be , where is some prior guess for the covariance matrix.
Properties
Log-expectation
The following formula plays a role in
variational Bayes
Variational Bayesian methods are a family of techniques for approximating intractable integrals arising in Bayesian inference and machine learning. They are typically used in complex statistical models consisting of observed variables (usually ...
derivations for
Bayes network
A Bayesian network (also known as a Bayes network, Bayes net, belief network, or decision network) is a probabilistic graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph (DAG). Bay ...
s
involving the Wishart distribution:
:
where
is the multivariate digamma function (the derivative of the log of the
multivariate gamma function).
Log-variance
The following variance computation could be of help in Bayesian statistics:
:
where
is the trigamma function. This comes up when computing the Fisher information of the Wishart random variable.
Entropy
The
information entropy
In information theory, the entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent to the variable's possible outcomes. Given a discrete random variable X, which takes values in the alphabet \ ...
of the distribution has the following formula:
:
where is the
normalizing constant of the distribution:
:
This can be expanded as follows:
:
Cross-entropy
The
cross entropy
In information theory, the cross-entropy between two probability distributions p and q over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set if a coding scheme used for the set is ...
of two Wishart distributions
with parameters
and
with parameters
is
:
Note that when
and
we recover the entropy.
KL-divergence
The
Kullback–Leibler divergence
In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted D_\text(P \parallel Q), is a type of statistical distance: a measure of how one probability distribution ''P'' is different fro ...
of
from
is
:
Characteristic function
The
characteristic function In mathematics, the term "characteristic function" can refer to any of several distinct concepts:
* The indicator function of a subset, that is the function
::\mathbf_A\colon X \to \,
:which for a given subset ''A'' of ''X'', has value 1 at point ...
of the Wishart distribution is
:
where denotes expectation. (Here is any matrix with the same dimensions as , indicates the identity matrix, and is a square root of ).
Properly interpreting this formula requires a little care, because noninteger complex powers are
multivalued; when is noninteger, the correct branch must be determined via
analytic continuation
In complex analysis, a branch of mathematics, analytic continuation is a technique to extend the domain of definition of a given analytic function. Analytic continuation often succeeds in defining further values of a function, for example in a ...
.
Theorem
If a random matrix has a Wishart distribution with degrees of freedom and variance matrix — write
— and is a matrix of
rank
Rank is the relative position, value, worth, complexity, power, importance, authority, level, etc. of a person or object within a ranking, such as:
Level or position in a hierarchical organization
* Academic rank
* Diplomatic rank
* Hierarchy
* H ...
, then
:
Corollary 1
If is a nonzero constant vector, then:
:
In this case,
is the
chi-squared distribution
In probability theory and statistics, the chi-squared distribution (also chi-square or \chi^2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables. The chi-square ...
and
(note that
is a constant; it is positive because is positive definite).
Corollary 2
Consider the case where (that is, the -th element is one and all others zero). Then corollary 1 above shows that
:
gives the marginal distribution of each of the elements on the matrix's diagonal.
George Seber points out that the Wishart distribution is not called the “multivariate chi-squared distribution” because the marginal distribution of the
off-diagonal element
In geometry, a diagonal is a line segment joining two vertices of a polygon or polyhedron, when those vertices are not on the same edge. Informally, any sloping line is called diagonal. The word ''diagonal'' derives from the ancient Greek δ ...
s is not chi-squared. Seber prefers to reserve the term
multivariate
Multivariate may refer to:
In mathematics
* Multivariable calculus
* Multivariate function
* Multivariate polynomial
In computing
* Multivariate cryptography
* Multivariate division algorithm
* Multivariate interpolation
* Multivariate optical c ...
for the case when all univariate marginals belong to the same family.
Estimator of the multivariate normal distribution
The Wishart distribution is the
sampling distribution of the
maximum-likelihood estimator (MLE) of the
covariance matrix
In probability theory and statistics, a covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix, or variance–covariance matrix) is a square Matrix (mathematics), matrix giving the covariance between ea ...
of a
multivariate normal distribution. A
derivation of the MLE uses the
spectral theorem
In mathematics, particularly linear algebra and functional analysis, a spectral theorem is a result about when a linear operator or matrix can be diagonalized (that is, represented as a diagonal matrix in some basis). This is extremely useful be ...
.
Bartlett decomposition
The Bartlett decomposition of a matrix from a -variate Wishart distribution with scale matrix and degrees of freedom is the factorization:
:
where is the
Cholesky factor of , and:
:
where
and independently. This provides a useful method for obtaining random samples from a Wishart distribution.
Marginal distribution of matrix elements
Let be a variance matrix characterized by
correlation coefficient
A correlation coefficient is a numerical measure of some type of correlation, meaning a statistical relationship between two variables. The variables may be two columns of a given data set of observations, often called a sample, or two component ...
and its lower Cholesky factor:
:
Multiplying through the Bartlett decomposition above, we find that a random sample from the Wishart distribution is
:
The diagonal elements, most evidently in the first element, follow the distribution with degrees of freedom (scaled by ) as expected. The off-diagonal element is less familiar but can be identified as a
normal variance-mean mixture In probability theory and statistics, a normal variance-mean mixture with mixing probability density g is the continuous probability distribution of a random variable Y of the form
:Y=\alpha + \beta V+\sigma \sqrtX,
where \alpha, \beta and \sigma ...
where the mixing density is a distribution. The corresponding marginal probability density for the off-diagonal element is therefore the
variance-gamma distribution
:
where is the
modified Bessel function of the second kind. Similar results may be found for higher dimensions, but the interdependence of the off-diagonal correlations becomes increasingly complicated. It is also possible to write down the
moment-generating function even in the ''noncentral'' case (essentially the ''n''th power of Craig (1936) equation 10) although the probability density becomes an infinite sum of Bessel functions.
The range of the shape parameter
It can be shown that the Wishart distribution can be defined if and only if the shape parameter belongs to the set
:
This set is named after Gindikin, who introduced it in the 1970s in the context of gamma distributions on homogeneous cones. However, for the new parameters in the discrete spectrum of the Gindikin ensemble, namely,
:
the corresponding Wishart distribution has no Lebesgue density.
Relationships to other distributions
* The Wishart distribution is related to the
inverse-Wishart distribution, denoted by
, as follows: If and if we do the change of variables , then
. This relationship may be derived by noting that the absolute value of the
Jacobian determinant of this change of variables is , see for example equation (15.15) in.
* In
Bayesian statistics
Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability where probability expresses a ''degree of belief'' in an event. The degree of belief may be based on prior knowledge about the event, ...
, the Wishart distribution is a
conjugate prior
In Bayesian probability theory, if the posterior distribution p(\theta \mid x) is in the same probability distribution family as the prior probability distribution p(\theta), the prior and posterior are then called conjugate distributions, and ...
for the
precision parameter of the
multivariate normal distribution, when the mean parameter is known.
* A generalization is the
multivariate gamma distribution.
* A different type of generalization is the
normal-Wishart distribution, essentially the product of a
multivariate normal distribution with a Wishart distribution.
See also
*
Chi-squared distribution
In probability theory and statistics, the chi-squared distribution (also chi-square or \chi^2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables. The chi-square ...
*
Complex Wishart distribution
*
F-distribution
*
Gamma distribution
*
Hotelling's T-squared distribution
*
Inverse-Wishart distribution
*
Multivariate gamma distribution
*
Student's t-distribution
*
Wilks' lambda distribution In statistics, Wilks' lambda distribution (named for Samuel S. Wilks), is a probability distribution used in multivariate hypothesis testing, especially with regard to the likelihood-ratio test and multivariate analysis of variance (MANOVA).
...
References
External links
A C++ library for random matrix generator
{{DEFAULTSORT:Wishart Distribution
Continuous distributions
Multivariate continuous distributions
Covariance and correlation
Random matrices
Conjugate prior distributions
Exponential family distributions