Multivariate normal distribution

In probability theory and statistics, the multivariate normal distribution, multivariate Gaussian distribution, or joint normal distribution is a generalization of the one-dimensional ( univariate) normal distribution to higher dimensions. One definition is that a random vector is said to be ''k''-variate normally distributed if every linear combination of its ''k'' components has a univariate normal distribution. Its importance derives mainly from the multivariate central limit theorem. The multivariate normal distribution is often used to describe, at least approximately, any set of (possibly) correlated real-valued random variables each of which clusters around a mean value.

Definitions

Notation and parameterization

The multivariate normal distribution of a ''k''-dimensional random vector $\mathbf = (X_1,\ldots,X_k)^$ can be written in the following notation:
: $\mathbf\ \sim\ \mathcal(\boldsymbol\mu,\, \boldsymbol\Sigma),$
or to make it explicitly known that ''X'' is ''k''-dimensional,
: $\mathbf\ \sim\ \mathcal_k(\boldsymbol\mu,\, \boldsymbol\Sigma),$
with ''k''-dimensional mean vector
: \boldsymbol\mu = \operatorname mathbf= ( \operatorname _1 \operatorname _2 \ldots, \operatorname _k) ^ \textbf,
and $k \times k$ covariance matrix
: \Sigma_ = \operatorname X_i - \mu_i)( X_j - \mu_j)= \operatorname _i, X_j
such that $1 \le i \le k$ and $1 \le j \le k$. The inverse of the covariance matrix is called the precision matrix, denoted by $\boldsymbol=\boldsymbol\Sigma^$.

Standard normal random vector

A real random vector $\mathbf = (X_1,\ldots,X_k)^$ is called a standard normal random vector if all of its components $X_i$ are independent and each is a zero-mean unit-variance normally distributed random variable, i.e. if $X_i \sim\ \mathcal(0,1)$ for all $i=1\ldots k$.

Centered normal random vector

A real random vector $\mathbf = (X_1,\ldots,X_k)^$ is called a centered normal random vector if there exists a deterministic $k \times \ell$ matrix $\boldsymbol$ such that $\boldsymbol \mathbf$ has the same distribution as $\mathbf$ where $\mathbf$ is a standard normal random vector with $\ell$ components.

Normal random vector

A real random vector $\mathbf = (X_1,\ldots,X_k)^$ is called a normal random vector if there exists a random $\ell$-vector $\mathbf$, which is a standard normal random vector, a $k$-vector $\mathbf$, and a $k \times \ell$ matrix $\boldsymbol$, such that $\mathbf=\boldsymbol \mathbf + \mathbf$.

Formally:

Here the covariance matrix is $\boldsymbol\Sigma = \boldsymbol \boldsymbol^$.

In the degenerate case where the covariance matrix is singular, the corresponding distribution has no density; see the section below for details. This case arises frequently in statistics; for example, in the distribution of the vector of residuals in the ordinary least squares regression. The $X_i$ are in general ''not'' independent; they can be seen as the result of applying the matrix $\boldsymbol$ to a collection of independent Gaussian variables $\mathbf$.

Equivalent definitions

The following definitions are equivalent to the definition given above. A random vector $\mathbf = (X_1, \ldots, X_k)^T$ has a multivariate normal distribution if it satisfies one of the following equivalent conditions.
*Every linear combination $Y=a_1 X_1 + \cdots + a_k X_k$ of its components is normally distributed. That is, for any constant vector $\mathbf \in \mathbb^k$, the random variable $Y=\mathbf^\mathbf$ has a univariate normal distribution, where a univariate normal distribution with zero variance is a point mass on its mean.
*There is a ''k''-vector $\mathbf$ and a symmetric, positive semidefinite $k \times k$ matrix $\boldsymbol\Sigma$, such that the characteristic function of $\mathbf$ is $\varphi_\mathbf(\mathbf) = \exp\Big( i\mathbf^T\boldsymbol\mu - \tfrac \mathbf^T\boldsymbol\Sigma \mathbf \Big).$

The spherical normal distribution can be characterised as the unique distribution where components are independent in any orthogonal coordinate system.

Density function

Non-degenerate case

The multivariate normal distribution is said to be "non-degenerate" when the symmetric covariance matrix $\boldsymbol\Sigma$ is positive definite. In this case the distribution has density

where $$ is a real ''k''-dimensional column vector and $, \boldsymbol\Sigma, \equiv \det\boldsymbol\Sigma$ is the determinant of $\boldsymbol\Sigma$, also known as the generalized variance. The equation above reduces to that of the univariate normal distribution if $\boldsymbol\Sigma$ is a $1 \times 1$ matrix (i.e. a single real number).

The circularly symmetric version of the complex normal distribution has a slightly different form.

Each iso-density locus — the locus of points in ''k''-dimensional space each of which gives the same particular value of the density — is an ellipse or its higher-dimensional generalization; hence the multivariate normal is a special case of the elliptical distributions.

The quantity $\sqrt$ is known as the Mahalanobis distance, which represents the distance of the test point $$ from the mean $$. Note that in the case when $k = 1$, the distribution reduces to a univariate normal distribution and the Mahalanobis distance reduces to the absolute value of the standard score. See also Interval below.

Bivariate case

In the 2-dimensional nonsingular case ($k = \operatorname\left(\Sigma\right) = 2$), the probability density function of a vector $\text$ is:

f(x,y) =
\frac
\exp
\left( -\frac\left \left(\frac\right)^2 -
2\rho\left(\frac\right)\left(\frac\right) +
\left(\frac\right)^2
\right \right)

where $\rho$ is the correlation between $X$ and $Y$ and
where $\sigma_X>0$ and $\sigma_Y>0$. In this case,
: $\boldsymbol\mu = \begin \mu_X \\ \mu_Y \end, \quad \boldsymbol\Sigma = \begin \sigma_X^2 & \rho \sigma_X \sigma_Y \\ \rho \sigma_X \sigma_Y & \sigma_Y^2 \end.$
In the bivariate case, the first equivalent condition for multivariate reconstruction of normality can be made less restrictive as it is sufficient to verify that countably many distinct linear combinations of $X$ and $Y$ are normal in order to conclude that the vector of $\text$ is bivariate normal.

The bivariate iso-density loci plotted in the $x,y$-plane are ellipses, whose principal axes are defined by the eigenvectors of the covariance matrix $\boldsymbol\Sigma$ (the major and minor semidiameters of the ellipse equal the square-root of the ordered eigenvalues).

As the absolute value of the correlation parameter $\rho$ increases, these loci are squeezed toward the following line :

: $y(x) = \sgn (\rho)\frac (x - \mu _X) + \mu_Y.$

This is because this expression, with $\sgn(\rho)$ (where sgn is the Sign function) replaced by $\rho$, is the best linear unbiased prediction of $Y$ given a value of $X$.

Degenerate case

If the covariance matrix $\boldsymbol\Sigma$ is not full rank, then the multivariate normal distribution is degenerate and does not have a density. More precisely, it does not have a density with respect to ''k''-dimensional Lebesgue measure (which is the usual measure assu