LogSumExp

The LogSumExp (LSE) (also called RealSoftMax or multivariable softplus) function is a smooth maximum – a smooth approximation to the maximum function, mainly used by machine learning algorithms. It is defined as the logarithm of the sum of the exponentials of the arguments:

$\mathrm(x_1, \dots, x_n) = \log\left( \exp(x_1) + \cdots + \exp(x_n) \right).$

Properties

The LogSumExp function domain is $\R^n$, the real coordinate space, and its codomain is $\R$, the real line.
It is an approximation to the maximum $\max_i x_i$ with the following bounds
$\max \leq \mathrm(x_1, \dots, x_n) \leq \max + \log(n).$
The first inequality is strict unless $n = 1$. The second inequality is strict unless all arguments are equal.
(Proof: Let $m = \max_i x_i$. Then $\exp(m) \leq \sum_^n \exp(x_i) \leq n \exp(m)$. Applying the logarithm to the inequality gives the result.)

In addition, we can scale the function to make the bounds tighter. Consider the function $\frac 1 t \mathrm(tx_1, \dots, tx_n)$. Then
$\max < \frac 1 t \mathrm(tx_1, \dots, tx_n) \leq \max + \frac.$
(Proof: Replace each $x_i$ with $tx_i$ for some $t>0$ in the inequalities above, to give
$\max < \mathrm(tx_1, \dots, tx_n)\leq \max + \log(n).$
and, since $t>0$
$t \max < \mathrm(tx_1, \dots, tx_n)\leq t\max + \log(n).$
finally, dividing by $t$ gives the result.)

Also, if we multiply by a negative number instead, we of course find a comparison to the $\min$ function:
$\min - \frac \leq \frac 1 \mathrm(-tx) < \min.$
The LogSumExp function is convex, and is strictly increasing everywhere in its domain (but not strictly convex everywhere).

Writing $\mathbf = (x_1, \dots, x_n),$ the partial derivatives are:
$\frac = \frac,$
which means the gradient of LogSumExp is the softmax function.

The convex conjugate of LogSumExp is the negative entropy.

log-sum-exp trick for log-domain calculations

The LSE function is often encountered when the usual arithmetic computations are performed on a logarithmic scale, as in log probability.

Similar to multiplication operations in linear-scale becoming simple additions in log-scale, an addition operation in
linear-scale becomes the LSE in log-scale:

$\mathrm(\log(x_1), ..., \log(x_n)) = \log(x_1 + \dots + x_n)$
A common purpose of using log-domain computations is to increase accuracy and avoid underflow and overflow problems
when very small or very large numbers are represented directly (i.e. in a linear domain) using limited-precision
floating point numbers.

Unfortunately, the use of LSE directly in this case can again cause overflow/underflow problems. Therefore, the
following equivalent must be used instead (especially when the accuracy of the above 'max' approximation is not sufficient).
Therefore, many math libraries such as IT++ provide a default routine of LSE and use this formula internally.

$\mathrm(x_1, \dots, x_n) = x^* + \log\left( \exp(x_1-x^*)+ \cdots + \exp(x_n-x^*) \right)$
where $x^* = \max$

A strictly convex log-sum-exp type function

LSE is convex but not strictly convex.
We can define a strictly convex log-sum-exp type function by adding an extra argument set to zero:

$\mathrm_0^+(x_1,...,x_n) = \mathrm(0,x_1,...,x_n)$
This function is a proper Bregman generator (strictly convex and differentiable).
It is encountered in machine learning, for example, as the cumulant of the multinomial/binomial family.

In tropical analysis, this is the sum in the log semiring.

See also

* Logarithmic mean
* Log semiring
* Smooth maximum
* Softmax function

References

{{refend

Logarithms