The data processing inequality is an
information theoretic concept that states that the information content of a signal cannot be increased via a local physical operation. This can be expressed concisely as 'post-processing cannot increase information'.
Statement
Let three random variables form the
Markov chain
In probability theory and statistics, a Markov chain or Markov process is a stochastic process describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. Informally ...
, implying that the conditional distribution of
depends only on
and is
conditionally independent of
. Specifically, we have such a Markov chain if the joint probability mass function can be written as
:
In this setting, no processing of
, deterministic or random, can increase the information that
contains about
. Using the
mutual information
In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual Statistical dependence, dependence between the two variables. More specifically, it quantifies the "Information conten ...
, this can be written as :
:
with the equality
if and only if
. That is,
and
contain the same information about
, and
also forms a Markov chain.
Proof
One can apply the
chain rule for mutual information to obtain two different decompositions of
:
:
By the relationship
, we know that
and
are conditionally independent, given
, which means the
conditional mutual information,
. The data processing inequality then follows from the non-negativity of
.
See also
*
Garbage in, garbage out
In computer science, garbage in, garbage out (GIGO) is the concept that flawed, biased or poor quality ("garbage") information or input (computer science), input produces a result or input/output, output of similar ("garbage") quality. The adage ...
References
External links
*http://www.scholarpedia.org/article/Mutual_information
Data processing
{{Comp-sci-stub