Projection (linear Algebra)
   HOME

TheInfoList



OR:

In
linear algebra Linear algebra is the branch of mathematics concerning linear equations such as: :a_1x_1+\cdots +a_nx_n=b, linear maps such as: :(x_1, \ldots, x_n) \mapsto a_1x_1+\cdots +a_nx_n, and their representations in vector spaces and through matrices. ...
and
functional analysis Functional analysis is a branch of mathematical analysis, the core of which is formed by the study of vector spaces endowed with some kind of limit-related structure (e.g. Inner product space#Definition, inner product, Norm (mathematics)#Defini ...
, a projection is a
linear transformation In mathematics, and more specifically in linear algebra, a linear map (also called a linear mapping, linear transformation, vector space homomorphism, or in some contexts linear function) is a mapping V \to W between two vector spaces that pre ...
P from a
vector space In mathematics and physics, a vector space (also called a linear space) is a set whose elements, often called ''vectors'', may be added together and multiplied ("scaled") by numbers called '' scalars''. Scalars are often real numbers, but can ...
to itself (an
endomorphism In mathematics, an endomorphism is a morphism from a mathematical object to itself. An endomorphism that is also an isomorphism is an automorphism. For example, an endomorphism of a vector space is a linear map , and an endomorphism of a g ...
) such that P\circ P=P. That is, whenever P is applied twice to any vector, it gives the same result as if it were applied once (i.e. P is
idempotent Idempotence (, ) is the property of certain operation (mathematics), operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application. The concept of idempotence ...
). It leaves its
image An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
unchanged. This definition of "projection" formalizes and generalizes the idea of
graphical projection A 3D projection (or graphical projection) is a design technique used to display a three-dimensional (3D) object on a two-dimensional (2D) surface. These projections rely on visual perspective and aspect analysis to project a complex object fo ...
. One can also consider the effect of a projection on a geometrical object by examining the effect of the projection on
point Point or points may refer to: Places * Point, Lewis, a peninsula in the Outer Hebrides, Scotland * Point, Texas, a city in Rains County, Texas, United States * Point, the NE tip and a ferry terminal of Lismore, Inner Hebrides, Scotland * Point ...
s in the object.


Definitions

A projection on a vector space V is a linear operator P : V \to V such that P^2 = P. When V has an
inner product In mathematics, an inner product space (or, rarely, a Hausdorff space, Hausdorff pre-Hilbert space) is a real vector space or a complex vector space with an operation (mathematics), operation called an inner product. The inner product of two ve ...
and is
complete Complete may refer to: Logic * Completeness (logic) * Completeness of a theory, the property of a theory that every formula in the theory's language or its negation is provable Mathematics * The completeness of the real numbers, which implies t ...
(i.e. when V is a
Hilbert space In mathematics, Hilbert spaces (named after David Hilbert) allow generalizing the methods of linear algebra and calculus from (finite-dimensional) Euclidean vector spaces to spaces that may be infinite-dimensional. Hilbert spaces arise natural ...
) the concept of
orthogonality In mathematics, orthogonality is the generalization of the geometric notion of ''perpendicularity''. By extension, orthogonality is also used to refer to the separation of specific features of a system. The term also has specialized meanings in ...
can be used. A projection P on a Hilbert space V is called an orthogonal projection if it satisfies \langle P \mathbf x, \mathbf y \rangle = \langle \mathbf x, P \mathbf y \rangle for all \mathbf x, \mathbf y \in V. A projection on a Hilbert space that is not orthogonal is called an oblique projection.


Projection matrix

* In the
finite-dimensional In mathematics, the dimension of a vector space ''V'' is the cardinality (i.e., the number of vectors) of a basis of ''V'' over its base field. p. 44, §2.36 It is sometimes called Hamel dimension (after Georg Hamel) or algebraic dimension to disti ...
case, a
square matrix In mathematics, a square matrix is a matrix with the same number of rows and columns. An ''n''-by-''n'' matrix is known as a square matrix of order Any two square matrices of the same order can be added and multiplied. Square matrices are often ...
P is called a projection matrix if it is equal to its square, i.e. if P^2 = P. * A square matrix P is called an orthogonal projection matrix if P^2 = P = P^ for a
real Real may refer to: Currencies * Brazilian real (R$) * Central American Republic real * Mexican real * Portuguese real * Spanish real * Spanish colonial real Music Albums * ''Real'' (L'Arc-en-Ciel album) (2000) * ''Real'' (Bright album) (2010) ...
matrix Matrix most commonly refers to: * ''The Matrix'' (franchise), an American media franchise ** ''The Matrix'', a 1999 science-fiction action film ** "The Matrix", a fictional setting, a virtual reality environment, within ''The Matrix'' (franchis ...
, and respectively P^2 = P = P^ for a
complex Complex commonly refers to: * Complexity, the behaviour of a system whose components interact in multiple ways so possible interactions are difficult to describe ** Complex system, a system composed of many components which may interact with each ...
matrix, where P^ denotes the
transpose In linear algebra, the transpose of a matrix is an operator which flips a matrix over its diagonal; that is, it switches the row and column indices of the matrix by producing another matrix, often denoted by (among other notations). The tr ...
of P and P^ denotes the adjoint or
Hermitian transpose In mathematics, the conjugate transpose, also known as the Hermitian transpose, of an m \times n complex matrix \boldsymbol is an n \times m matrix obtained by transposing \boldsymbol and applying complex conjugate on each entry (the complex con ...
of P. * A projection matrix that is not an orthogonal projection matrix is called an oblique projection matrix. The
eigenvalue In linear algebra, an eigenvector () or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted b ...
s of a projection matrix must be 0 or 1.


Examples


Orthogonal projection

For example, the function which maps the point (x,y,z) in three-dimensional space \mathbb^3 to the point (x,y,0) is an orthogonal projection onto the ''xy''-plane. This function is represented by the matrix P = \begin 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end. The action of this matrix on an arbitrary
vector Vector most often refers to: *Euclidean vector, a quantity with a magnitude and a direction *Vector (epidemiology), an agent that carries and transmits an infectious pathogen into another living organism Vector may also refer to: Mathematic ...
is P \begin x \\ y \\ z \end = \begin x \\ y \\ 0 \end. To see that P is indeed a projection, i.e., P = P^2, we compute P^2 \begin x \\ y \\ z \end = P \begin x \\ y \\ 0 \end = \begin x \\ y \\ 0 \end = P\begin x \\ y \\ z \end. Observing that P^ = P shows that the projection is an orthogonal projection.


Oblique projection

A simple example of a non-orthogonal (oblique) projection is P = \begin 0 & 0 \\ \alpha & 1 \end. Via
matrix multiplication In mathematics, particularly in linear algebra, matrix multiplication is a binary operation that produces a matrix from two matrices. For matrix multiplication, the number of columns in the first matrix must be equal to the number of rows in the s ...
, one sees that P^2 = \begin 0 & 0 \\ \alpha & 1 \end \begin 0 & 0 \\ \alpha & 1 \end = \begin 0 & 0 \\ \alpha & 1 \end = P. showing that P is indeed a projection. The projection P is orthogonal
if and only if In logic and related fields such as mathematics and philosophy, "if and only if" (shortened as "iff") is a biconditional logical connective between statements, where either both statements are true or both are false. The connective is bicondi ...
\alpha = 0 because only then P^ = P.


Properties and classification


Idempotence

By definition, a projection P is
idempotent Idempotence (, ) is the property of certain operation (mathematics), operations in mathematics and computer science whereby they can be applied multiple times without changing the result beyond the initial application. The concept of idempotence ...
(i.e. P^2 = P).


Open map

Every projection is an
open map In mathematics, more specifically in topology, an open map is a function between two topological spaces that maps open sets to open sets. That is, a function f : X \to Y is open if for any open set U in X, the image f(U) is open in Y. Likewise, a ...
, meaning that it maps each
open set In mathematics, open sets are a generalization of open intervals in the real line. In a metric space (a set along with a distance defined between any two points), open sets are the sets that, with every point , contain all points that are suf ...
in the
domain Domain may refer to: Mathematics *Domain of a function, the set of input values for which the (total) function is defined **Domain of definition of a partial function **Natural domain of a partial function **Domain of holomorphy of a function * Do ...
to an open set in the
subspace topology In topology and related areas of mathematics, a subspace of a topological space ''X'' is a subset ''S'' of ''X'' which is equipped with a topology induced from that of ''X'' called the subspace topology (or the relative topology, or the induced to ...
of the
image An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
. That is, for any vector \mathbf and any ball B_\mathbf (with positive radius) centered on \mathbf, there exists a ball B_ (with positive radius) centered on P\mathbf that is wholly contained in the image P(B_\mathbf).


Complementarity of image and kernel

Let W be a finite-dimensional vector space and P be a projection on W. Suppose the subspaces U and V are the
image An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
and
kernel Kernel may refer to: Computing * Kernel (operating system), the central component of most operating systems * Kernel (image processing), a matrix used for image convolution * Compute kernel, in GPGPU programming * Kernel method, in machine learnin ...
of P respectively. Then P has the following properties: # P is the
identity operator Identity may refer to: * Identity document * Identity (philosophy) * Identity (social science) * Identity (mathematics) Arts and entertainment Film and television * ''Identity'' (1987 film), an Iranian film * ''Identity'' (2003 film), a ...
I on U: \forall \mathbf x \in U: P \mathbf x = \mathbf x. # We have a
direct sum The direct sum is an operation between structures in abstract algebra, a branch of mathematics. It is defined differently, but analogously, for different kinds of structures. To see how the direct sum is used in abstract algebra, consider a more ...
W = U \oplus V. Every vector \mathbf x \in W may be decomposed uniquely as \mathbf x = \mathbf u + \mathbf v with \mathbf u = P \mathbf x and \mathbf v = \mathbf x - P \mathbf x = \left(I-P\right) \mathbf x, and where \mathbf u \in U, \mathbf v \in V. The image and kernel of a projection are ''complementary'', as are P and Q = I - P. The operator Q is also a projection as the image and kernel of P become the kernel and image of Q and vice versa. We say P is a projection along V onto U (kernel/image) and Q is a projection along U onto V.


Spectrum

In infinite-dimensional vector spaces, the
spectrum A spectrum (plural ''spectra'' or ''spectrums'') is a condition that is not limited to a specific set of values but can vary, without gaps, across a continuum. The word was first used scientifically in optics to describe the rainbow of colors i ...
of a projection is contained in \ as (\lambda I - P)^ = \frac 1 \lambda I + \frac 1 P. Only 0 or 1 can be an
eigenvalue In linear algebra, an eigenvector () or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted b ...
of a projection. This implies that an orthogonal projection P is always a
positive semi-definite matrix In mathematics, a symmetric matrix M with real entries is positive-definite if the real number z^\textsfMz is positive for every nonzero real column vector z, where z^\textsf is the transpose of More generally, a Hermitian matrix (that is, a co ...
. In general, the corresponding
eigenspace In linear algebra, an eigenvector () or characteristic vector of a linear transformation is a nonzero vector that changes at most by a scalar factor when that linear transformation is applied to it. The corresponding eigenvalue, often denoted b ...
s are (respectively) the kernel and range of the projection. Decomposition of a vector space into direct sums is not unique. Therefore, given a subspace V, there may be many projections whose range (or kernel) is V. If a projection is nontrivial it has minimal polynomial x^2 - x = x (x-1), which factors into distinct linear factors, and thus P is
diagonalizable In linear algebra, a square matrix A is called diagonalizable or non-defective if it is similar to a diagonal matrix, i.e., if there exists an invertible matrix P and a diagonal matrix D such that or equivalently (Such D are not unique.) F ...
.


Product of projections

The product of projections is not in general a projection, even if they are orthogonal. If two projections
commute Commute, commutation or commutative may refer to: * Commuting, the process of travelling between a place of residence and a place of work Mathematics * Commutative property, a property of a mathematical operation whose result is insensitive to th ...
then their product is a projection, but the
converse Converse may refer to: Mathematics and logic * Converse (logic), the result of reversing the two parts of a definite or implicational statement ** Converse implication, the converse of a material implication ** Converse nonimplication, a logical c ...
is false: the product of two non-commuting projections may be a projection. If two orthogonal projections commute then their product is an orthogonal projection. If the product of two orthogonal projections is an orthogonal projection, then the two orthogonal projections commute (more generally: two self-adjoint
endomorphism In mathematics, an endomorphism is a morphism from a mathematical object to itself. An endomorphism that is also an isomorphism is an automorphism. For example, an endomorphism of a vector space is a linear map , and an endomorphism of a g ...
s commute if and only if their product is self-adjoint).


Orthogonal projections

When the vector space W has an
inner product In mathematics, an inner product space (or, rarely, a Hausdorff space, Hausdorff pre-Hilbert space) is a real vector space or a complex vector space with an operation (mathematics), operation called an inner product. The inner product of two ve ...
and is complete (is a
Hilbert space In mathematics, Hilbert spaces (named after David Hilbert) allow generalizing the methods of linear algebra and calculus from (finite-dimensional) Euclidean vector spaces to spaces that may be infinite-dimensional. Hilbert spaces arise natural ...
) the concept of
orthogonality In mathematics, orthogonality is the generalization of the geometric notion of ''perpendicularity''. By extension, orthogonality is also used to refer to the separation of specific features of a system. The term also has specialized meanings in ...
can be used. An orthogonal projection is a projection for which the range U and the null space V are orthogonal subspaces. Thus, for every \mathbf x and \mathbf y in W, \langle P \mathbf x, (\mathbf y - P \mathbf y) \rangle = \langle (\mathbf x - P \mathbf x) , P \mathbf y \rangle = 0. Equivalently: \langle \mathbf x, P \mathbf y \rangle = \langle P \mathbf x, P \mathbf y \rangle = \langle P \mathbf x, \mathbf y \rangle. A projection is orthogonal if and only if it is
self-adjoint In mathematics, and more specifically in abstract algebra, an element ''x'' of a *-algebra is self-adjoint if x^*=x. A self-adjoint element is also Hermitian, though the reverse doesn't necessarily hold. A collection ''C'' of elements of a st ...
. Using the self-adjoint and idempotent properties of P, for any \mathbf x and \mathbf y in W we have P\mathbf \in U, \mathbf - P\mathbf \in V, and \langle P \mathbf x, \mathbf y - P \mathbf y \rangle = \langle P^2 \mathbf x, \mathbf y - P \mathbf y \rangle = \langle P \mathbf x, P \left(I-P\right) \mathbf y \rangle = \langle P \mathbf x, \left(P-P^2\right) \mathbf y \rangle = 0 where \langle \cdot, \cdot \rangle is the inner product associated with W. Therefore, P and I - P are orthogonal projections. The other direction, namely that if P is orthogonal then it is self-adjoint, follows from \langle \mathbf x, P \mathbf y \rangle = \langle P \mathbf x, \mathbf y \rangle = \langle \mathbf x, P^* \mathbf y \rangle for every x and y in W; thus P=P^*.


Properties and special cases

An orthogonal projection is a
bounded operator In functional analysis and operator theory, a bounded linear operator is a linear transformation L : X \to Y between topological vector spaces (TVSs) X and Y that maps bounded subsets of X to bounded subsets of Y. If X and Y are normed vector s ...
. This is because for every \mathbf v in the vector space we have, by the
Cauchy–Schwarz inequality The Cauchy–Schwarz inequality (also called Cauchy–Bunyakovsky–Schwarz inequality) is considered one of the most important and widely used inequalities in mathematics. The inequality for sums was published by . The corresponding inequality fo ...
: \left \, P \mathbf v\right\, ^2 = \langle P \mathbf v, P \mathbf v \rangle = \langle P \mathbf v, \mathbf v \rangle \leq \left\, P \mathbf v\right\, \cdot \left\, \mathbf v\right\, Thus \left\, P \mathbf v\right\, \leq \left\, \mathbf v\right\, . For finite-dimensional complex or real vector spaces, the standard inner product can be substituted for \langle \cdot, \cdot \rangle.


=Formulas

= A simple case occurs when the orthogonal projection is onto a line. If \mathbf u is a
unit vector In mathematics, a unit vector in a normed vector space is a vector (often a spatial vector) of length 1. A unit vector is often denoted by a lowercase letter with a circumflex, or "hat", as in \hat (pronounced "v-hat"). The term ''direction vecto ...
on the line, then the projection is given by the
outer product In linear algebra, the outer product of two coordinate vector In linear algebra, a coordinate vector is a representation of a vector as an ordered list of numbers (a tuple) that describes the vector in terms of a particular ordered basis. An ea ...
P_\mathbf = \mathbf u \mathbf u^\mathsf. (If \mathbf u is complex-valued, the transpose in the above equation is replaced by a Hermitian transpose). This operator leaves u invariant, and it annihilates all vectors orthogonal to \mathbf u, proving that it is indeed the orthogonal projection onto the line containing u. A simple way to see this is to consider an arbitrary vector \mathbf x as the sum of a component on the line (i.e. the projected vector we seek) and another perpendicular to it, \mathbf x = \mathbf x_\parallel + \mathbf x_\perp. Applying projection, we get P_ \mathbf x = \mathbf u \mathbf u^\mathsf \mathbf x_\parallel + \mathbf u \mathbf u^\mathsf \mathbf x_\perp = \mathbf u \left( \sgn\left(\mathbf u^\mathsf \mathbf x_\parallel\right) \left \, \mathbf x_\parallel \right \, \right) + \mathbf u \cdot \mathbf 0 = \mathbf x_\parallel by the properties of the
dot product In mathematics, the dot product or scalar productThe term ''scalar product'' means literally "product with a scalar as a result". It is also used sometimes for other symmetric bilinear forms, for example in a pseudo-Euclidean space. is an algebra ...
of parallel and perpendicular vectors. This formula can be generalized to orthogonal projections on a subspace of arbitrary
dimension In physics and mathematics, the dimension of a Space (mathematics), mathematical space (or object) is informally defined as the minimum number of coordinates needed to specify any Point (geometry), point within it. Thus, a Line (geometry), lin ...
. Let \mathbf u_1, \ldots, \mathbf u_k be an
orthonormal basis In mathematics, particularly linear algebra, an orthonormal basis for an inner product space ''V'' with finite dimension is a basis for V whose vectors are orthonormal, that is, they are all unit vectors and orthogonal to each other. For example, ...
of the subspace U, with the assumption that the integer k \geq 1, and let A denote the n \times k matrix whose columns are \mathbf u_1, \ldots, \mathbf u_k, i.e., A = \begin \mathbf u_1 & \cdots & \mathbf u_k \end. Then the projection is given by: P_A = A A^\mathsf which can be rewritten as P_A = \sum_i \langle \mathbf u_i, \cdot \rangle \mathbf u_i. The matrix A^\mathsf is the
partial isometry In functional analysis a partial isometry is a linear map between Hilbert spaces such that it is an isometry on the orthogonal complement of its kernel. The orthogonal complement of its kernel is called the initial subspace and its range is called ...
that vanishes on the
orthogonal complement In the mathematical fields of linear algebra and functional analysis, the orthogonal complement of a subspace ''W'' of a vector space ''V'' equipped with a bilinear form ''B'' is the set ''W''⊥ of all vectors in ''V'' that are orthogonal to every ...
of U and A is the isometry that embeds U into the underlying vector space. The range of P_A is therefore the ''final space'' of A. It is also clear that A A^ is the identity operator on U. The orthonormality condition can also be dropped. If \mathbf u_1, \ldots, \mathbf u_k is a (not necessarily orthonormal)
basis Basis may refer to: Finance and accounting * Adjusted basis, the net cost of an asset after adjusting for various tax-related items *Basis point, 0.01%, often used in the context of interest rates * Basis trading, a trading strategy consisting ...
with k \geq 1, and A is the matrix with these vectors as columns, then the projection is: P_A = A \left(A^\mathsf A\right)^ A^\mathsf. The matrix A still embeds U into the underlying vector space but is no longer an isometry in general. The matrix \left(A^\mathsfA\right)^ is a "normalizing factor" that recovers the norm. For example, the
rank Rank is the relative position, value, worth, complexity, power, importance, authority, level, etc. of a person or object within a ranking, such as: Level or position in a hierarchical organization * Academic rank * Diplomatic rank * Hierarchy * H ...
-1 operator \mathbf u \mathbf u^\mathsf is not a projection if \left\, \mathbf u \right\, \neq 1. After dividing by \mathbf u^\mathsf \mathbf u = \left\, \mathbf u \right\, ^2, we obtain the projection \mathbf u \left(\mathbf u^\mathsf \mathbf u \right)^ \mathbf u^\mathsf onto the subspace spanned by u. In the general case, we can have an arbitrary
positive definite In mathematics, positive definiteness is a property of any object to which a bilinear form or a sesquilinear form may be naturally associated, which is positive-definite. See, in particular: * Positive-definite bilinear form * Positive-definite f ...
matrix D defining an inner product \langle x, y \rangle_D = y^\dagger Dx, and the projection P_A is given by P_A x = \operatorname_ \left\, x - y\right\, ^2_D. Then P_A = A \left(A^\mathsf D A\right)^ A^\mathsf D. When the range space of the projection is generated by a
frame A frame is often a structural system that supports other components of a physical construction and/or steel frame that limits the construction's extent. Frame and FRAME may also refer to: Physical objects In building construction *Framing (con ...
(i.e. the number of generators is greater than its dimension), the formula for the projection takes the form: P_A = A A^+. Here A^+ stands for the Moore–Penrose pseudoinverse. This is just one of many ways to construct the projection operator. If \begin A & B \end is a non-singular matrix and A^\mathsfB = 0 (i.e., B is the
null space In mathematics, the kernel of a linear map, also known as the null space or nullspace, is the linear subspace of the Domain of a function, domain of the map which is mapped to the zero vector. That is, given a linear map between two vector space ...
matrix of A), the following holds: \begin I &= \begin A & B \end \begin A & B \end^\begin A^\mathsf \\ B^\mathsf \end^ \begin A^\mathsf \\ B^\mathsf \end \\ &= \begin A & B \end \left( \begin A^\mathsf \\ B^\mathsf \end \begin A & B \end \right )^ \begin A^\mathsf \\B^\mathsf \end \\ &= \begin A & B \end \beginA^\mathsfA&O\\O&B^\mathsfB\end^ \begin A^\mathsf \\ B^\mathsf \end\\ pt &= A \left(A^\mathsfA\right)^ A^\mathsf + B \left(B^\mathsfB\right)^ B^\mathsf \end If the orthogonal condition is enhanced to A^\mathsfW B = A^\mathsfW^\mathsfB = 0 with W non-singular, the following holds: I = \beginA & B\end \begin\left(A^\mathsf W A\right)^ A^\mathsf \\ \left(B^\mathsf W B\right)^ B^\mathsf \end W. All these formulas also hold for complex inner product spaces, provided that the
conjugate transpose In mathematics, the conjugate transpose, also known as the Hermitian transpose, of an m \times n complex matrix \boldsymbol is an n \times m matrix obtained by transposing \boldsymbol and applying complex conjugate on each entry (the complex con ...
is used instead of the transpose. Further details on sums of projectors can be found in Banerjee and Roy (2014). Also see Banerjee (2004) for application of sums of projectors in basic
spherical trigonometry Spherical trigonometry is the branch of spherical geometry that deals with the metrical relationships between the sides and angles of spherical triangles, traditionally expressed using trigonometric functions. On the sphere, geodesics are gr ...
.


Oblique projections

The term ''oblique projections'' is sometimes used to refer to non-orthogonal projections. These projections are also used to represent spatial figures in two-dimensional drawings (see
oblique projection Oblique projection is a simple type of technical drawing of graphical projection used for producing two-dimensional (2D) images of three-dimensional (3D) objects. The objects are not in perspective and so do not correspond to any view of an ...
), though not as frequently as orthogonal projections. Whereas calculating the fitted value of an
ordinary least squares In statistics, ordinary least squares (OLS) is a type of linear least squares method for choosing the unknown parameters in a linear regression model (with fixed level-one effects of a linear function of a set of explanatory variables) by the prin ...
regression requires an orthogonal projection, calculating the fitted value of an instrumental variables regression requires an oblique projection. Projections are defined by their null space and the basis vectors used to characterize their range (which is the complement of the null space). When these basis vectors are orthogonal to the null space, then the projection is an orthogonal projection. When these basis vectors are not orthogonal to the null space, the projection is an oblique projection, or just a general projection.


A matrix representation formula for a nonzero projection operator

Let P be a linear operator P : V \to V such that P^2 = P and assume that P : V \to V is not the zero operator. Let the vectors \mathbf u_1, \ldots, \mathbf u_k form a basis for the range of the projection, and assemble these vectors in the n \times k matrix A. Therefore the integer k \geq 1, otherwise k = 0 and P is the zero operator. The range and the null space are complementary spaces, so the null space has dimension n - k. It follows that the
orthogonal complement In the mathematical fields of linear algebra and functional analysis, the orthogonal complement of a subspace ''W'' of a vector space ''V'' equipped with a bilinear form ''B'' is the set ''W''⊥ of all vectors in ''V'' that are orthogonal to every ...
of the null space has dimension k. Let \mathbf v_1, \ldots, \mathbf v_k form a basis for the orthogonal complement of the null space of the projection, and assemble these vectors in the matrix B. Then the projection P (with the condition k \geq 1) is given by P = A \left(B^\mathsf A\right)^ B^\mathsf. This expression generalizes the formula for orthogonal projections given above. A standard proof of this expression is the following. For any vector \mathbf x in the vector space V, we can decompose \mathbf = \mathbf_1 + \mathbf_2, where vector \mathbf_1 = P(\mathbf) is in the image of P, and vector \mathbf_2 = \mathbf - P(\mathbf). So P(\mathbf_2) = P(\mathbf) - P^2(\mathbf)= \mathbf, and then \mathbf_2 is in the null space of P. In other words, the vector \mathbf_1 is in the column space of A, so \mathbf_1 = A \mathbf for some k dimension vector \mathbf and the vector \mathbf_2 satisfies B^\mathsf \mathbf_2=\mathbf by the construction of B. Put these conditions together, and we find a vector \mathbf so that B^\mathsf (\mathbf-A\mathbf)=\mathbf. Since matrices A and B are of full rank k by their construction, the k\times k-matrix B^\mathsf A is invertible. So the equation B^\mathsf (\mathbf-A\mathbf)=\mathbf gives the vector \mathbf= (B^A)^ B^ \mathbf. In this way, P\mathbf = \mathbf_1 = A\mathbf= A(B^A)^ B^ \mathbf for any vector \mathbf \in V and hence P = A(B^A)^ B^. In the case that P is an orthogonal projection, we can take A = B, and it follows that P=A \left(A^\mathsf A\right)^ A^\mathsf. By using this formula, one can easily check that P=P^\mathsf. In general, if the vector space is over complex number field, one then uses the
Hermitian transpose In mathematics, the conjugate transpose, also known as the Hermitian transpose, of an m \times n complex matrix \boldsymbol is an n \times m matrix obtained by transposing \boldsymbol and applying complex conjugate on each entry (the complex con ...
A^* and has the formula P=A \left(A^* A\right)^ A^*. Recall that one can define the
Moore–Penrose inverse In mathematics, and in particular linear algebra, the Moore–Penrose inverse of a matrix is the most widely known generalization of the inverse matrix. It was independently described by E. H. Moore in 1920, Arne Bjerhammar in 1951, and Roger Pe ...
of the matrix A by A^= (A^*A)^A^* since A has full column rank, so P=A A^.


Singular Values

Note that I-P is also an oblique projection. The singular values of P and I-P can be computed by an
orthonormal basis In mathematics, particularly linear algebra, an orthonormal basis for an inner product space ''V'' with finite dimension is a basis for V whose vectors are orthonormal, that is, they are all unit vectors and orthogonal to each other. For example, ...
of A. Let Q_A be an orthonormal basis of A and let Q_A^ be the
orthogonal complement In the mathematical fields of linear algebra and functional analysis, the orthogonal complement of a subspace ''W'' of a vector space ''V'' equipped with a bilinear form ''B'' is the set ''W''⊥ of all vectors in ''V'' that are orthogonal to every ...
of Q_A. Denote the singular values of the matrix Q_A^T A (B^T A)^ B^T Q_A^ by the positive values \gamma_1 \ge \gamma_2 \ge \ldots \ge \gamma_k . With this, the singular values for P are: \sigma_i = \begin \sqrt & 1 \le i \le k \\ 0 & \text \end and the singular values for I-P are \sigma_i = \begin \sqrt & 1 \le i \le k \\ 1 & k+1 \le i \le n-k \\ 0 & \text \end This implies that the largest singular values of P and I-P are equal, and thus that the
matrix norm In mathematics, a matrix norm is a vector norm in a vector space whose elements (vectors) are matrices (of given dimensions). Preliminaries Given a field K of either real or complex numbers, let K^ be the -vector space of matrices with m rows ...
of the oblique projections are the same. However, the
condition number In numerical analysis, the condition number of a function measures how much the output value of the function can change for a small change in the input argument. This is used to measure how sensitive a function is to changes or errors in the input ...
satisfies the relation \kappa(I-P) = \frac \ge \frac = \kappa(P), and is therefore not necessarily equal.


Finding projection with an inner product

Let V be a vector space (in this case a plane) spanned by orthogonal vectors \mathbf u_1, \mathbf u_2, \dots, \mathbf u_p. Let y be a vector. One can define a projection of \mathbf y onto V as \operatorname_V \mathbf y = \frac \mathbf u^i where repeated indices are summed over ( Einstein sum notation). The vector \mathbf y can be written as an orthogonal sum such that \mathbf y = \operatorname_V \mathbf y + \mathbf z. \operatorname_V \mathbf y is sometimes denoted as \hat. There is a theorem in linear algebra that states that this \mathbf z is the smallest distance (the ''
orthogonal distance In geometry, the perpendicular distance between two objects is the distance from one to the other, measured along a line that is perpendicular to one or both. The distance from a point to a line is the distance to the nearest point on that line. Th ...
'') from \mathbf y to V and is commonly used in areas such as
machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...
.


Canonical forms

Any projection P=P^2 on a vector space of dimension d over a
field Field may refer to: Expanses of open ground * Field (agriculture), an area of land used for agricultural purposes * Airfield, an aerodrome that lacks the infrastructure of an airport * Battlefield * Lawn, an area of mowed grass * Meadow, a grass ...
is a
diagonalizable matrix In linear algebra, a square matrix A is called diagonalizable or non-defective if it is similar to a diagonal matrix, i.e., if there exists an invertible matrix P and a diagonal matrix D such that or equivalently (Such D are not unique.) ...
, since its minimal polynomial divides x^2-x, which splits into distinct linear factors. Thus there exists a basis in which P has the form :P = I_r\oplus 0_ where r is the
rank Rank is the relative position, value, worth, complexity, power, importance, authority, level, etc. of a person or object within a ranking, such as: Level or position in a hierarchical organization * Academic rank * Diplomatic rank * Hierarchy * H ...
of P. Here I_r is the
identity matrix In linear algebra, the identity matrix of size n is the n\times n square matrix with ones on the main diagonal and zeros elsewhere. Terminology and notation The identity matrix is often denoted by I_n, or simply by I if the size is immaterial o ...
of size r, and 0_ is the
zero matrix In mathematics, particularly linear algebra, a zero matrix or null matrix is a matrix all of whose entries are zero. It also serves as the additive identity of the additive group of m \times n matrices, and is denoted by the symbol O or 0 followed ...
of size d-r. If the vector space is complex and equipped with an
inner product In mathematics, an inner product space (or, rarely, a Hausdorff space, Hausdorff pre-Hilbert space) is a real vector space or a complex vector space with an operation (mathematics), operation called an inner product. The inner product of two ve ...
, then there is an ''orthonormal'' basis in which the matrix of ''P'' is :P = \begin1&\sigma_1 \\ 0&0\end \oplus \cdots \oplus \begin1&\sigma_k \\ 0&0\end \oplus I_m \oplus 0_s. where \sigma_1 \geq \sigma_2\geq \dots \geq \sigma_k > 0. The
integer An integer is the number zero (), a positive natural number (, , , etc.) or a negative integer with a minus sign (−1, −2, −3, etc.). The negative numbers are the additive inverses of the corresponding positive numbers. In the language ...
s k,s,m and the real numbers \sigma_i are uniquely determined. Note that 2k+s+m=d. The factor I_m \oplus 0_s corresponds to the maximal invariant subspace on which P acts as an ''orthogonal'' projection (so that ''P'' itself is orthogonal if and only if k=0) and the \sigma_i-blocks correspond to the ''oblique'' components.


Projections on normed vector spaces

When the underlying vector space X is a (not necessarily finite-dimensional)
normed vector space In mathematics, a normed vector space or normed space is a vector space over the real or complex numbers, on which a norm is defined. A norm is the formalization and the generalization to real vector spaces of the intuitive notion of "length" i ...
, analytic questions, irrelevant in the finite-dimensional case, need to be considered. Assume now X is a
Banach space In mathematics, more specifically in functional analysis, a Banach space (pronounced ) is a complete normed vector space. Thus, a Banach space is a vector space with a metric that allows the computation of vector length and distance between vector ...
. Many of the algebraic results discussed above survive the passage to this context. A given direct sum decomposition of X into complementary subspaces still specifies a projection, and vice versa. If X is the direct sum X = U \oplus V, then the operator defined by P(u+v) = u is still a projection with range U and kernel V. It is also clear that P^2 = P. Conversely, if P is projection on X, i.e. P^2 = P, then it is easily verified that (1-P)^2 = (1-P). In other words, 1 - P is also a projection. The relation P^2 = P implies 1 = P + (1-P) and X is the direct sum \operatorname(P) \oplus \operatorname(1 - P). However, in contrast to the finite-dimensional case, projections need not be
continuous Continuity or continuous may refer to: Mathematics * Continuity (mathematics), the opposing concept to discreteness; common examples include ** Continuous probability distribution or random variable in probability and statistics ** Continuous ...
in general. If a subspace U of X is not closed in the norm topology, then the projection onto U is not continuous. In other words, the range of a continuous projection P must be a closed subspace. Furthermore, the kernel of a continuous projection (in fact, a continuous linear operator in general) is closed. Thus a ''continuous'' projection P gives a decomposition of X into two complementary ''closed'' subspaces: X = \operatorname(P) \oplus \ker(P) = \ker(1-P) \oplus \ker(P). The converse holds also, with an additional assumption. Suppose U is a closed subspace of X. If there exists a closed subspace V such that , then the projection P with range U and kernel V is continuous. This follows from the
closed graph theorem In mathematics, the closed graph theorem may refer to one of several basic results characterizing continuous functions in terms of their graphs. Each gives conditions when functions with closed graphs are necessarily continuous. Graphs and m ...
. Suppose and . One needs to show that Px=y. Since U is closed and , ''y'' lies in U, i.e. . Also, . Because V is closed and , we have x-y \in V, i.e. P(x-y)=Px-Py=Px-y=0, which proves the claim. The above argument makes use of the assumption that both U and V are closed. In general, given a closed subspace U, there need not exist a complementary closed subspace V, although for
Hilbert space In mathematics, Hilbert spaces (named after David Hilbert) allow generalizing the methods of linear algebra and calculus from (finite-dimensional) Euclidean vector spaces to spaces that may be infinite-dimensional. Hilbert spaces arise natural ...
s this can always be done by taking the
orthogonal complement In the mathematical fields of linear algebra and functional analysis, the orthogonal complement of a subspace ''W'' of a vector space ''V'' equipped with a bilinear form ''B'' is the set ''W''⊥ of all vectors in ''V'' that are orthogonal to every ...
. For Banach spaces, a one-dimensional subspace always has a closed complementary subspace. This is an immediate consequence of
Hahn–Banach theorem The Hahn–Banach theorem is a central tool in functional analysis. It allows the extension of bounded linear functionals defined on a subspace of some vector space to the whole space, and it also shows that there are "enough" continuous linear f ...
. Let U be the linear span of u. By Hahn–Banach, there exists a bounded
linear functional In mathematics, a linear form (also known as a linear functional, a one-form, or a covector) is a linear map from a vector space to its field of scalars (often, the real numbers or the complex numbers). If is a vector space over a field , the s ...
\varphi such that . The operator P(x)=\varphi(x)u satisfies P^2=P, i.e. it is a projection. Boundedness of \varphi implies continuity of P and therefore \ker(P) = \operatorname(I-P) is a closed complementary subspace of U.


Applications and further considerations

Projections (orthogonal and otherwise) play a major role in
algorithm In mathematics and computer science, an algorithm () is a finite sequence of rigorous instructions, typically used to solve a class of specific Computational problem, problems or to perform a computation. Algorithms are used as specificat ...
s for certain linear algebra problems: *
QR decomposition In linear algebra, a QR decomposition, also known as a QR factorization or QU factorization, is a decomposition of a matrix ''A'' into a product ''A'' = ''QR'' of an orthogonal matrix ''Q'' and an upper triangular matrix ''R''. QR decompo ...
(see
Householder transformation In linear algebra, a Householder transformation (also known as a Householder reflection or elementary reflector) is a linear transformation that describes a reflection about a plane or hyperplane containing the origin. The Householder transformati ...
and Gram–Schmidt decomposition); *
Singular value decomposition In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix. It generalizes the eigendecomposition of a square normal matrix with an orthonormal eigenbasis to any \ m \times n\ matrix. It is related ...
* Reduction to Hessenberg form (the first step in many
eigenvalue algorithm In numerical analysis, one of the most important problems is designing efficient and stable algorithms for finding the eigenvalues of a matrix. These eigenvalue algorithms may also find eigenvectors. Eigenvalues and eigenvectors Given an square ...
s) *
Linear regression In statistics, linear regression is a linear approach for modelling the relationship between a scalar response and one or more explanatory variables (also known as dependent and independent variables). The case of one explanatory variable is call ...
* Projective elements of matrix algebras are used in the construction of certain K-groups in
Operator K-theory In mathematics, operator K-theory is a noncommutative analogue of topological K-theory for Banach algebras with most applications used for C*-algebras. Overview Operator K-theory resembles topological K-theory more than algebraic K-theory. In pa ...
As stated above, projections are a special case of idempotents. Analytically, orthogonal projections are non-commutative generalizations of characteristic functions. Idempotents are used in classifying, for instance,
semisimple algebra In ring theory, a branch of mathematics, a semisimple algebra is an associative artinian algebra over a field which has trivial Jacobson radical (only the zero element of the algebra is in the Jacobson radical). If the algebra is finite-dimensio ...
s, while
measure theory In mathematics, the concept of a measure is a generalization and formalization of geometrical measures ( length, area, volume) and other common notions, such as mass and probability of events. These seemingly distinct concepts have many simil ...
begins with considering characteristic functions of
measurable set In mathematics, the concept of a measure is a generalization and formalization of geometrical measures (length, area, volume) and other common notions, such as mass and probability of events. These seemingly distinct concepts have many simila ...
s. Therefore, as one can imagine, projections are very often encountered in the context of
operator algebra In functional analysis, a branch of mathematics, an operator algebra is an algebra of continuous linear operators on a topological vector space, with the multiplication given by the composition of mappings. The results obtained in the study of ...
s. In particular, a
von Neumann algebra In mathematics, a von Neumann algebra or W*-algebra is a *-algebra of bounded operators on a Hilbert space that is closed in the weak operator topology and contains the identity operator. It is a special type of C*-algebra. Von Neumann algeb ...
is generated by its complete
lattice Lattice may refer to: Arts and design * Latticework, an ornamental criss-crossed framework, an arrangement of crossing laths or other thin strips of material * Lattice (music), an organized grid model of pitch ratios * Lattice (pastry), an orna ...
of projections.


Generalizations

More generally, given a map between normed vector spaces T\colon V \to W, one can analogously ask for this map to be an isometry on the orthogonal complement of the kernel: that (\ker T)^\perp \to W be an isometry (compare
Partial isometry In functional analysis a partial isometry is a linear map between Hilbert spaces such that it is an isometry on the orthogonal complement of its kernel. The orthogonal complement of its kernel is called the initial subspace and its range is called ...
); in particular it must be
onto In mathematics, a surjective function (also known as surjection, or onto function) is a function that every element can be mapped from element so that . In other words, every element of the function's codomain is the image of one element of i ...
. The case of an orthogonal projection is when ''W'' is a subspace of ''V.'' In
Riemannian geometry Riemannian geometry is the branch of differential geometry that studies Riemannian manifolds, smooth manifolds with a ''Riemannian metric'', i.e. with an inner product on the tangent space at each point that varies smoothly from point to poin ...
, this is used in the definition of a
Riemannian submersion In differential geometry, a branch of mathematics, a Riemannian submersion is a submersion from one Riemannian manifold to another that respects the metrics, meaning that it is an orthogonal projection on tangent spaces. Formal definition Let (' ...
.


See also

*
Centering matrix In mathematics and multivariate statistics, the centering matrixJohn I. Marden, ''Analyzing and Modeling Rank Data'', Chapman & Hall, 1995, , page 59. is a symmetric and idempotent matrix, which when multiplied with a vector has the same effect a ...
, which is an example of a projection matrix. *
Dykstra's projection algorithm Dykstra's algorithm is a method that computes a point in the intersection of convex sets, and is a variant of the alternating projection method (also called the projections onto convex sets method). In its simplest form, the method finds a point ...
to compute the projection onto an intersection of sets *
Invariant subspace In mathematics, an invariant subspace of a linear mapping ''T'' : ''V'' → ''V '' i.e. from some vector space ''V'' to itself, is a subspace ''W'' of ''V'' that is preserved by ''T''; that is, ''T''(''W'') ⊆ ''W''. General descrip ...
*
Least-squares spectral analysis Least-squares spectral analysis (LSSA) is a method of estimating a frequency spectrum, based on a least squares fit of sinusoids to data samples, similar to Fourier analysis. Fourier analysis, the most used spectral method in science, generally ...
*
Orthogonalization In linear algebra, orthogonalization is the process of finding a set of orthogonal vectors that span a particular subspace. Formally, starting with a linearly independent set of vectors in an inner product space (most commonly the Euclidean spa ...
* Properties of trace


Notes


References

* * *


External links

* , from MIT OpenCourseWare * , by Pavel Grinfeld.
Planar Geometric Projections Tutorial
– a simple-to-follow tutorial explaining the different types of planar geometric projections. {{linear algebra Functional analysis Linear algebra Linear operators