In
statistics
Statistics (from German language, German: ', "description of a State (polity), state, a country") is the discipline that concerns the collection, organization, analysis, interpretation, and presentation of data. In applying statistics to a s ...
and in particular in
regression analysis, a design matrix, also known as model matrix or regressor matrix and often denoted by X, is a
matrix
Matrix (: matrices or matrixes) or MATRIX may refer to:
Science and mathematics
* Matrix (mathematics), a rectangular array of numbers, symbols or expressions
* Matrix (logic), part of a formula in prenex normal form
* Matrix (biology), the m ...
of values of
explanatory variable
A variable is considered dependent if it depends on (or is hypothesized to depend on) an independent variable. Dependent variables are studied under the supposition or demand that they depend, by some law or rule (e.g., by a mathematical function ...
s of a set of objects. Each row represents an individual object, with the successive columns corresponding to the variables and their specific values for that object. The design matrix is used in certain
statistical model
A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of Sample (statistics), sample data (and similar data from a larger Statistical population, population). A statistical model repre ...
s, e.g., the
general linear model
The general linear model or general multivariate regression model is a compact way of simultaneously writing several multiple linear regression models. In that sense it is not a separate statistical linear model. The various multiple linear regre ...
. It can contain
indicator variables (ones and zeros) that indicate group membership in an
ANOVA, or it can contain values of
continuous variable
In mathematics and statistics, a quantitative variable (mathematics), variable may be continuous or discrete. If it can take on two real number, real values and all the values between them, the variable is continuous in that Interval (mathemati ...
s.
The design matrix contains data on the
independent variable
A variable is considered dependent if it depends on (or is hypothesized to depend on) an independent variable. Dependent variables are studied under the supposition or demand that they depend, by some law or rule (e.g., by a mathematical function ...
s (also called explanatory variables), in a statistical model that is intended to explain observed data on a response variable (often called a
dependent variable
A variable is considered dependent if it depends on (or is hypothesized to depend on) an independent variable. Dependent variables are studied under the supposition or demand that they depend, by some law or rule (e.g., by a mathematical functio ...
). The theory relating to such models uses the design matrix as input to some
linear algebra
Linear algebra is the branch of mathematics concerning linear equations such as
:a_1x_1+\cdots +a_nx_n=b,
linear maps such as
:(x_1, \ldots, x_n) \mapsto a_1x_1+\cdots +a_nx_n,
and their representations in vector spaces and through matrix (mathemat ...
: see for example
linear regression
In statistics, linear regression is a statistical model, model that estimates the relationship between a Scalar (mathematics), scalar response (dependent variable) and one or more explanatory variables (regressor or independent variable). A mode ...
. A notable feature of the concept of a design matrix is that it is able to represent a number of different
experimental design
The design of experiments (DOE), also known as experiment design or experimental design, is the design of any task that aims to describe and explain the variation of information under conditions that are hypothesized to reflect the variation. ...
s and statistical models, e.g.,
ANOVA,
ANCOVA, and linear regression.
Definition
The design matrix is defined to be a matrix
such that
(the ''j''
th column of the ''i''
th row of
) represents the value of the ''j''
th variable associated with the ''i''
th object.
A regression model may be represented via matrix multiplication as
:
where ''X'' is the design matrix,
is a vector of the model's coefficients (one for each variable),
is a vector of random errors with mean zero, and ''y'' is the vector of predicted outputs for each object.
Size
The design matrix has dimension ''n''-by-''p'', where ''n'' is the number of samples observed, and ''p'' is the number of variables (
features) measured in all samples.
In this representation different rows typically represent different repetitions of an experiment, while columns represent different types of data (say, the results from particular probes). For example, suppose an experiment is run where 10 people are pulled off the street and asked 4 questions. The data matrix ''M'' would be a 10×4 matrix (meaning 10 rows and 4 columns). The datum in row ''i'' and column ''j'' of this matrix would be the answer of the ''i''
th person to the ''j''
th question.
Examples
Arithmetic mean
The design matrix for an
arithmetic mean
In mathematics and statistics, the arithmetic mean ( ), arithmetic average, or just the ''mean'' or ''average'' is the sum of a collection of numbers divided by the count of numbers in the collection. The collection is often a set of results fr ...
is a
column
A column or pillar in architecture and structural engineering is a structural element that transmits, through compression, the weight of the structure above to other structural elements below. In other words, a column is a compression member ...
vector of ones
In mathematics, a matrix of ones or all-ones matrix is a matrix with every entry equal to one. For example:
:J_2 = \begin
1 & 1 \\
1 & 1
\end,\quad
J_3 = \begin
1 & 1 & 1 \\
1 & 1 & 1 \\
1 & 1 & 1
\end,\quad
J_ = \begin
1 & 1 & 1 & 1 & 1 \\
1 ...
.
Simple linear regression
This section gives an example of
simple linear regression
In statistics, simple linear regression (SLR) is a linear regression model with a single explanatory variable. That is, it concerns two-dimensional sample points with one independent variable and one dependent variable (conventionally, the ''x ...
—that is, regression with only a single explanatory variable—with seven observations.
The seven data points are , for ''i'' = 1, 2, …, 7. The simple linear regression model is
:
where
is the ''y''-intercept and
is the slope of the regression line. This model can be represented in matrix form as
:
where the first column of 1s in the design matrix allows estimation of the ''y''-intercept while the second column contains the ''x''-values associated with the corresponding ''y''-values. The matrix whose columns are 1's and ''xs in this example is the design matrix.
Multiple regression
This section contains an example of
multiple regression with two covariates (explanatory variables): ''w'' and ''x''.
Again suppose that the data consist of seven observations, and that for each observed value to be predicted (
), values ''w''
''i'' and ''x''
''i'' of the two covariates are also observed. The model to be considered is
:
This model can be written in matrix terms as
:
Here the 7×3 matrix on the right side is the design matrix.
One-way ANOVA (cell means model)
This section contains an example with a one-way analysis of variance (
ANOVA) with three groups and seven observations. The given data set has the first three observations belonging to the first group, the following two observations belonging to the second group and the final two observations belonging to the third group.
If the model to be fit is just the mean of each group, then the model is
:
which can be written
:
In this model
represents the mean of the
th group.
One-way ANOVA (offset from reference group)
The ANOVA model could be equivalently written as each group parameter
being an offset from some overall reference. Typically this reference point is taken to be one of the groups under consideration. This makes sense in the context of comparing multiple treatment groups to a control group and the control group is considered the "reference". In this example, group 1 was chosen to be the reference group. As such the model to be fit is
:
with the constraint that
is zero.
:
In this model
is the mean of the reference group and
is the difference from group
to the reference group.
is not included in the matrix because its difference from the reference group (itself) is necessarily zero.
See also
*
Moment matrix
*
Projection matrix
In statistics, the projection matrix (\mathbf), sometimes also called the influence matrix or hat matrix (\mathbf), maps the vector of response values (dependent variable values) to the vector of fitted values (or predicted values). It describes ...
*
Jacobian matrix and determinant
In vector calculus, the Jacobian matrix (, ) of a vector-valued function of several variables is the matrix of all its first-order partial derivatives. If this matrix is square, that is, if the number of variables equals the number of compon ...
*
Scatter matrix
*
Gram matrix
*
Vandermonde matrix
References
Further reading
*
{{Matrix classes
Matrices (mathematics)
Regression analysis
Design of experiments
Multivariate statistics
Data