Differential entropy (also referred to as continuous entropy) is a concept in
information theory
Information theory is the scientific study of the quantification, storage, and communication of information. The field was originally established by the works of Harry Nyquist and Ralph Hartley, in the 1920s, and Claude Shannon in the 1940s. ...
that began as an attempt by
Claude Shannon
Claude Elwood Shannon (April 30, 1916 – February 24, 2001) was an American mathematician, electrical engineer, and cryptographer known as a "father of information theory".
As a 21-year-old master's degree student at the Massachusetts I ...
to extend the idea of (Shannon)
entropy
Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodyna ...
, a measure of average
surprisal of a
random variable
A random variable (also called random quantity, aleatory variable, or stochastic variable) is a mathematical formalization of a quantity or object which depends on random events. It is a mapping or a function from possible outcomes (e.g., the po ...
, to continuous
probability distribution
In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a random phenomenon ...
s. Unfortunately, Shannon did not derive this formula, and rather just assumed it was the correct continuous analogue of discrete entropy, but it is not. The actual continuous version of discrete entropy is the
limiting density of discrete points
In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy.
It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential e ...
(LDDP). Differential entropy (described here) is commonly encountered in the literature, but it is a limiting case of the LDDP, and one that loses its fundamental association with discrete
entropy
Entropy is a scientific concept, as well as a measurable physical property, that is most commonly associated with a state of disorder, randomness, or uncertainty. The term and the concept are used in diverse fields, from classical thermodyna ...
.
In terms of
measure theory
In mathematics, the concept of a measure is a generalization and formalization of geometrical measures (length, area, volume) and other common notions, such as mass and probability of events. These seemingly distinct concepts have many simila ...
, the differential entropy of a
probability measure
In mathematics, a probability measure is a real-valued function defined on a set of events in a probability space that satisfies measure properties such as ''countable additivity''. The difference between a probability measure and the more ge ...
is the negative
relative entropy from that measure to the
Lebesgue measure
In measure theory, a branch of mathematics, the Lebesgue measure, named after French mathematician Henri Lebesgue, is the standard way of assigning a measure to subsets of ''n''-dimensional Euclidean space. For ''n'' = 1, 2, or 3, it coincides wi ...
, where the latter is treated as if it were a probability measure, despite being unnormalized.
Definition
Let
be a random variable with a
probability density function
In probability theory, a probability density function (PDF), or density of a continuous random variable, is a function whose value at any given sample (or point) in the sample space (the set of possible values taken by the random variable) ca ...
whose
support
Support may refer to:
Arts, entertainment, and media
* Supporting character
Business and finance
* Support (technical analysis)
* Child support
* Customer support
* Income Support
Construction
* Support (structure), or lateral support, a ...
is a set
. The ''differential entropy''
or
is defined as
For probability distributions which don't have an explicit density function expression, but have an explicit
quantile function
In probability and statistics, the quantile function, associated with a probability distribution of a random variable, specifies the value of the random variable such that the probability of the variable being less than or equal to that value equ ...
expression,
, then
can be defined in terms of the derivative of
i.e. the quantile density function
as
:
.
As with its discrete analog, the units of differential entropy depend on the base of the
logarithm
In mathematics, the logarithm is the inverse function to exponentiation. That means the logarithm of a number to the base is the exponent to which must be raised, to produce . For example, since , the ''logarithm base'' 10 ...
, which is usually 2 (i.e., the units are
bits). See
logarithmic units
A logarithmic scale (or log scale) is a way of displaying numerical data over a very wide range of values in a compact way—typically the largest numbers in the data are hundreds or even thousands of times larger than the smallest numbers. Such a ...
for logarithms taken in different bases. Related concepts such as
joint
A joint or articulation (or articular surface) is the connection made between bones, ossicles, or other hard structures in the body which link an animal's skeletal system into a functional whole.Saladin, Ken. Anatomy & Physiology. 7th ed. McGraw- ...
,
conditional
Conditional (if then) may refer to:
*Causal conditional, if X then Y, where X is a cause of Y
*Conditional probability, the probability of an event A given that another event B has occurred
*Conditional proof, in logic: a proof that asserts a co ...
differential entropy, and
relative entropy are defined in a similar fashion. Unlike the discrete analog, the differential entropy has an offset that depends on the units used to measure
.
For example, the differential entropy of a quantity measured in millimeters will be more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of more than the same quantity divided by 1000.
One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the
uniform distribution
Uniform distribution may refer to:
* Continuous uniform distribution
* Discrete uniform distribution
* Uniform distribution (ecology)
* Equidistributed sequence
See also
*
* Homogeneous distribution
In mathematics, a homogeneous distribution ...
has ''negative'' differential entropy
:
being less than that of
which has ''zero'' differential entropy. Thus, differential entropy does not share all properties of discrete entropy.
Note that the continuous
mutual information
In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables. More specifically, it quantifies the " amount of information" (in units such ...
has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of ''partitions'' of
and
as these partitions become finer and finer. Thus it is invariant under non-linear
homeomorphisms
In the mathematical field of topology, a homeomorphism, topological isomorphism, or bicontinuous function is a bijective and continuous function between topological spaces that has a continuous inverse function. Homeomorphisms are the isomo ...
(continuous and uniquely invertible maps), including linear
transformations of
and
, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.
For the direct analogue of discrete entropy extended to the continuous space, see
limiting density of discrete points
In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy.
It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential e ...
.
Properties of differential entropy
* For probability densities
and
, the
Kullback–Leibler divergence
In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy and I-divergence), denoted D_\text(P \parallel Q), is a type of statistical distance: a measure of how one probability distribution ''P'' is different fr ...
is greater than or equal to 0 with equality only if
almost everywhere. Similarly, for two random variables
and
,
and
with equality
if and only if
In logic and related fields such as mathematics and philosophy, "if and only if" (shortened as "iff") is a biconditional logical connective between statements, where either both statements are true or both are false.
The connective is bic ...
and
are
independent
Independent or Independents may refer to:
Arts, entertainment, and media Artist groups
* Independents (artist group), a group of modernist painters based in the New Hope, Pennsylvania, area of the United States during the early 1930s
* Independe ...
.
* The chain rule for differential entropy holds as in the discrete case
::
.
* Differential entropy is translation invariant, i.e. for a constant
.
::
* Differential entropy is in general not invariant under arbitrary invertible maps.
:: In particular, for a constant
:::
:: For a vector valued random variable
and an invertible (square)
matrix
Matrix most commonly refers to:
* ''The Matrix'' (franchise), an American media franchise
** '' The Matrix'', a 1999 science-fiction action film
** "The Matrix", a fictional setting, a virtual reality environment, within ''The Matrix'' (franchi ...
:::
* In general, for a transformation from a random vector to another random vector with same dimension
, the corresponding entropies are related via
::
:where
is the
Jacobian
In mathematics, a Jacobian, named for Carl Gustav Jacob Jacobi, may refer to:
* Jacobian matrix and determinant
* Jacobian elliptic functions
* Jacobian variety
*Intermediate Jacobian
In mathematics, the intermediate Jacobian of a compact Kähle ...
of the transformation
. The above inequality becomes an equality if the transform is a bijection. Furthermore, when
is a rigid rotation, translation, or combination thereof, the Jacobian determinant is always 1, and
.
* If a random vector
has mean zero and
covariance
In probability theory and statistics, covariance is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the le ...
matrix
,