Continuous Bernoulli Distribution

In probability theory, statistics, and machine learning, the continuous Bernoulli distribution is a family of continuous probability distributions parameterized by a single shape parameter $\lambda \in (0, 1)$, defined on the unit interval $x \in$, 1/math>, by:
:$p(x , \lambda) \propto \lambda^x (1-\lambda)^.$

The continuous Bernoulli distribution arises in deep learning and computer vision, specifically in the context of variational autoencoders, for modeling the pixel intensities of natural images. As such, it defines a proper probabilistic counterpart for the commonly used binary cross entropy loss, which is often applied to continuous, ,1/math>-valued data. This practice amounts to ignoring the normalizing constant of the continuous Bernoulli distribution, since the binary cross entropy loss only defines a true log-likelihood for discrete, $\$-valued data.

The continuous Bernoulli also defines an exponential family of distributions. Writing $\eta = \log\left(\lambda/(1-\lambda)\right)$ for the natural parameter, the density can be rewritten in canonical form:
$p(x , \eta) \propto \exp (\eta x)$.

Related distributions

Bernoulli distribution

The continuous Bernoulli can be thought of as a continuous relaxation of the Bernoulli distribution, which is defined on the discrete set $\$ by the probability mass function:
:$p(x) = p^x (1-p)^,$
where $p$ is a scalar parameter between 0 and 1. Applying this same functional form on the continuous interval ,1 results in the continuous Bernoulli probability density function, up to a normalizing constant.

Beta distribution

The Beta distribution has the density function:
:$p(x) \propto x^ (1-x)^,$
which can be re-written as:
:$p(x) \propto x_1^ x_2^,$
where $\alpha_1, \alpha_2$ are positive scalar parameters, and $(x_1, x_2)$ represents an arbitrary point inside the 1-simplex, $\Delta^ = \$. Switching the role of the parameter and the argument in this density function, we obtain:
:$p(x) \propto \alpha_1^ \alpha_2^.$
This family is only identifiable up to the linear constraint $\alpha_1 + \alpha_2 = 1$, whence we obtain:
:$p(x) \propto \lambda^ (1-\lambda)^,$
corresponding exactly to the continuous Bernoulli density.

Exponential distribution

An exponential distribution restricted to the unit interval is equivalent to a continuous Bernoulli distribution with appropriate parameter.

Continuous categorical distribution

The multivariate generalization of the continuous Bernoulli is called the continuous-categorical.Gordon-Rodriguez, E., Loaiza-Ganem, G., & Cunningham, J. P. (2020). The continuous categorical: a novel simplex-valued exponential family. In 36th International Conference on Machine Learning, ICML 2020. International Machine Learning Society (IMLS).

References

{{Probability distributions

Continuous distributions
Exponential family distributions