Bayes Classifier

In statistical classification, the Bayes classifier minimizes the probability of misclassification.

Definition

Suppose a pair $(X,Y)$ takes values in $\mathbb^d \times \$, where $Y$ is the class label of $X$. Assume that the conditional distribution of ''X'', given that the label ''Y'' takes the value ''r'' is given by

:$(X\mid Y=r) \sim P_r$ for $r=1,2,\dots,K$

where "$\sim$" means "is distributed as", and where $P_r$ denotes a probability distribution.

A classifier is a rule that assigns to an observation ''X''=''x'' a guess or estimate of what the unobserved label ''Y''=''r'' actually was. In theoretical terms, a classifier is a measurable function $C: \mathbb^d \to \$, with the interpretation that ''C'' classifies the point ''x'' to the class ''C''(''x''). The probability of misclassification, or risk, of a classifier ''C'' is defined as
:$\mathcal(C) = \operatorname\.$

The Bayes classifier is
:$C^\text(x) = \underset \operatorname(Y=r \mid X=x).$

In practice, as in most of statistics, the difficulties and subtleties are associated with modeling the probability distributions effectively—in this case, $\operatorname(Y=r \mid X=x)$. The Bayes classifier is a useful benchmark in statistical classification.

The excess risk of a general classifier $C$ (possibly depending on some training data) is defined as $\mathcal(C) - \mathcal(C^\text).$
Thus this non-negative quantity is important for assessing the performance of different classification techniques. A classifier is said to be consistent if the excess risk converges to zero as the size of the training data set tends to infinity.

Considering the components $x_i$ of $x$ to be mutually independent, we get the naive bayes classifier, where $C^\text(x) = \underset \operatorname(Y=r)\prod_^P_r(x).$

Proof of Optimality

Proof that the Bayes classifier is optimal and Bayes error rate is minimal proceeds as follows.

Define the variables: Risk $R(h)$, Bayes risk $R^*$, all possible classes to which the points can be classified $Y = \$. Let the posterior probability of a point belonging to class 1 be $\eta(x)=Pr(Y=1, X=x)$. Define the classifier $\mathcal^*$as

$\mathcal^*(x)=\begin1&,\eta(x)\geqslant 0.5\\ 0&,\eta(x)<0.5\end$

Then we have the following results:

(a) $R(h^*)=R^*$, i.e. $h^*$ is a Bayes classifier,

(b) For any classifier $h$, the ''excess risk'' satisfies R(h)-R^*=2\mathbb_X\left min(\eta(X),1-.html" ;"title="\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X 2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>\eta(x)-0.5, \cdot \mathbb_\right/math>

(c) R^* = \mathbb_X\left min(\eta(X),1-\eta(X))\right/math>

Proof of (a): For any classifier $h$, we have

R(h) = \mathbb_\left \mathbb_ \right

=\mathbb_X\mathbb_ mathbb_ /math> (due to Fubini's theorem)

= \mathbb_X eta(X)\mathbb_ +(1-\eta(X))\mathbb_

Notice that $R(h)$ is minimised by taking $\forall x\in X$,

$h(x)=\begin1&,\eta(x)\geqslant 1-\eta(x)\\ 0&,\text\end$

Therefore the minimum possible risk is the Bayes risk, $R^*= R(h^*)$.

Proof of (b):

\begin
R(h)-R^* &= R(h)-R(h^*)\\
&= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_-\eta(X)\mathbb_-(1-\eta(X))\mathbb_\
&=\mathbb_X 2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification>2\eta(X)-1, \mathbb_\
&= 2\mathbb_X \eta(X)-0.5, \mathbb_\end

Proof of (c):

\begin
R(h^*) &= \mathbb_X eta(X)\mathbb_+(1-\eta(X))\mathbb_\
&= \mathbb_X min(\eta(X),1-\eta(X))\end

The general case that the Bayes classifier minimises classification error when each element can belong to either of ''n'' categories proceeds by towering expectations as follows.

\begin
\mathbb_Y(\mathbb_) &= \mathbb_X\mathbb_\left(\mathbb_, X=x\right)\\
&= \mathbb\left X=x)\mathbb_+Pr(Y=2, X=x)\mathbb_+\dots+Pr(Y=n, X=x)\mathbb_\right
\end

This is minimised by simultaneously minimizing all the terms of the expectation using the classifier$h(x)=k,\quad \arg\max_Pr(Y=k, X=x)$

for each observation ''x''.

See also

*Naive Bayes classifier

References

{{reflist, 1

Bayesian statistics
Statistical classification