vanishing gradient problem

In machine learning, the vanishing gradient problem is encountered when training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, during each iteration of training each of the neural network's weights receives an update proportional to the partial derivative of the error function with respect to the current weight. The problem is that in some cases, the gradient will be vanishingly small, effectively preventing the weight from changing its value. In the worst case, this may completely stop the neural network from further training. As one example of the problem cause, traditional activation functions such as the hyperbolic tangent function have gradients in the range , and backpropagation computes gradients by the chain rule. This has the effect of multiplying of these small numbers to compute gradients of the early layers in an -layer network, meaning that the gradient (error signal) decreases exponentially with while the early layers train very slowly.

Back-propagation allowed researchers to train supervised deep artificial neural networks from scratch, initially with little success. Hochreiter's diplom thesis of 1991 formally identified the reason for this failure in the "vanishing gradient problem", which not only affects many-layered feedforward networks, but also recurrent networks. The latter are trained by unfolding them into very deep feedforward networks, where a new layer is created for each time step of an input sequence processed by the network. (The combination of unfolding and backpropagation is termed backpropagation through time.)

When activation functions are used whose derivatives can take on larger values, one risks encountering the related exploding gradient problem.

Prototypical models

This section is based on.

Recurrent network model

A generic recurrent network has hidden states $h_1, h_2, ...$ inputs $u_1, u_2, ...$, and outputs $x_1, x_2, ...$. Let it be parametrized by $\theta$, so that the system evolves as$(h_t, x_t) = F(h_, u_t, \theta)$Often, the output $x_t$ is a function of $h_t$, as some $x_t = G(h_t)$. The vanishing gradient problem already presents itself clearly when $x_t = h_t$, so we identify them. This gives us a recurrent network with $x_t = F(x_, u_t, \theta)$Now, take its differential:$\begin dx_t &= \nabla_\theta F(x_, u_t, \theta) d\theta + \nabla_x F(x_, u_t, \theta) dx_ \\ &= \nabla_\theta F(x_, u_t, \theta) d\theta + \nabla_x F(x_, u_t, \theta)(\nabla_\theta F(x_, u_, \theta) d\theta + \nabla_x F(x_, u_, \theta) dx_) \\ &= \cdots \\ &= \left(\nabla_\theta F(x_, u_t, \theta) + \nabla_x F(x_, u_t, \theta) \nabla_\theta F(x_, u_, \theta) + \cdots \right) d\theta \end$Training the network requires us to define a loss function to be minimized. Let it be $L(x_T, u_1, ..., u_T)$, then minimizing it by gradient descent gives\Delta \theta = -\eta \cdot\left \nabla_x L(x_T)\left(\nabla_\theta F(x_, u_t, \theta) + \nabla_x F(x_, u_t, \theta) \nabla_\theta F(x_, u_, \theta) + \cdots \right)
\rightT where $\eta$ is the learning rate.

The vanishing/exploding gradient problem appears because there are repeated multiplications, of the form$\nabla_x F(x_, u_t, \theta) \nabla_x F(x_, u_, \theta)\nabla_x F(x_, u_, \theta) \cdots$

Example: recurrent network with sigmoid activation

For a concrete example, consider a typical recurrent network defined by

$x_t = F(x_, u_t, \theta) = W_ \sigma(x_) + W_ u_t + b$where $\theta = (W_, W_)$ is the network parameter, $\sigma$ is the sigmoid activation function, applied to each vector coordinate separately, and $b$ is the bias vector.

Then, $\nabla_x F(x_, u_t, \theta) = W_ \mathop(\sigma'(x_))$, and so $\begin \nabla_x F(x_, u_t, \theta) & \nabla_x F(x_, u_, \theta)\cdots \nabla_x F(x_, u_, \theta) \\ = W_ \mathop(\sigma'(x_)) & W_ \mathop(\sigma'(x_)) \cdots W_ \mathop(\sigma'(x_)) \end$Since $, \sigma', \leq 1$, the operator norm of the above multiplication is bounded above by $\, W_\, ^k$. So if the spectral radius of $W_$ is $\gamma<1$, then at large $k$, the above multiplication has operator norm bounded above by $\gamma^k \to 0$. This is the prototypical vanishing gradient problem.

The effect of a vanishing gradient is that the network cannot learn long-range effects. Recall Equation ():$\nabla_\theta L = \nabla_x L(x_T, u_1, ..., u_T)\left(\nabla_\theta F(x_, u_t, \theta) + \nabla_x F(x_, u_t, \theta) \nabla_\theta F(x_, u_, \theta) + \cdots \right)$The components of $\nabla_\theta F(x, u, \theta)$ are just components of $\sigma(x)$ and $u$, so if $u_t, u_, ...$ are bounded, then $\, \nabla _F(x_,u_,\theta)\,$ is also bounded by some $M>0$, and so the terms in $\nabla_\theta L$ decay as $M\gamma^k$. This means that, effectively, $\nabla_\theta L$ is affected only by the first $O(\gamma^)$ terms in the sum.

If $\gamma\geq 1$, the above analysis does not quite work. For the prototypical exploding gradient problem, the next model is clearer.

Dynamical systems model

Following (Doya, 1993), consider this one-neuron recurrent network with sigmoid activation:$x_ = (1-\epsilon) x_t + \epsilon \sigma(wx_t + b) + \epsilon w' u_t$At the small $\epsilon$ limit, the dynamics of the network becomes$\frac = -x(t) + \sigma(wx(t)+b) + w' u(t)$Consider first the autonomous case, with $u=0$. Set $w=5.0$, and vary $b$ in 3, -2. As $b$ decreases, the system has 1 stable point, then has 2 stable points and 1 unstable point, and finally has 1 stable point again. Explicitly, the stable points are $(x, b) = \left(x, \ln\left(\frac\right)-5x \right)$.

Now consider $\frac$ and $\frac$, where $T$ is large enough that the system has settled into one of the stable points.

If $(x(0), b)$ puts the system very close to an unstable point, then a tiny variation in $x(0)$ or $b$ would make $x(T)$ move from one stable point to the other. This makes $\frac$ and $\frac$ both very large, a case of the exploding gradient.

If $(x(0), b)$ puts the system far from an unstable point, then a small variation in $x(0)$ would have no effect on $x(T)$, making $\frac= 0$, a case of the vanishing gradient.

Note that in this case, $\frac\approx \frac = \left(\frac-5\right)^$ neither decays to zero nor blows up to infinity. Indeed, it's the only well-behaved gradient, which explains why early researches focused on learning or designing recurrent networks systems that could perform long-ranged computations (such as outputting the first input it sees at the very end of an episode) by shaping its stable attractors.

For the general case, the intuition still holds. The situation is plotted in, Figures 3, 4, and 5.

Geometric model

Continue using the above one-neuron network, fixing $w = 5, x(0) = 0.5, u(t) = 0$, and consider a loss function defined by $L(x(T)) = (0.855 - x(T))^2$. This produces a rather pathological loss landscape: as $b$ approach $-2.5$ from above, the loss approaches zero, but as soon as $b$ crosses $-2.5$, the attractor basin changes, and loss jumps to 0.50.

Consequently, attempting to train $b$ by gradient descent would "hit a wall in the loss landscape", and cause exploding gradient. A slightly more complex situation is plotted in, Figures 6.

Solutions

To overcome this problem, several methods were proposed.

Batch normalization

Batch normalization is a standard method for solving both the exploding and the vanishing gradient problems.

Gradient clipping

recommends clipping the norm of $\nabla_\theta L$ by $\epsilon$:$g = \begin \nabla_\theta L \quad \text \, \nabla_\theta L\, < \epsilon,\\ \epsilon \cdot \frac\quad \text \end$where $\epsilon$ is the "threshold" hyperparameter, the maximum norm that the gradient is allowed to reach. Simply clipping each entry of $\nabla_\theta L$ separately by $\epsilon$ works as well in practice.

This does not solve the vanishing gradient problem.

Multi-level hierarchy

One is Jürgen Schmidhuber's multi-level hierarchy of networks (1992) pre-trained one level at a time through unsupervised learning, fine-tuned through backpropagation.J. Schmidhuber., "Learning complex, extended sequences using the principle of history compression," ''Neural Computation'', 4, pp. 234–242, 1992. Here each level learns a compressed representation of the observations that is fed to the next level.

Related approach

Similar ideas have been used in feed-forward neural networks for unsupervised pre-training to structure a neural network, making it first learn generally useful feature detectors. Then the network is trained further by supervised backpropagation to classify labeled data. The deep belief network model by Hinton et al. (2006) involves learning the distribution of a high level representation using successive layers of binary or real-valued latent variables. It uses a restricted Boltzmann machine to model each new layer of higher level features. Each new layer guarantees an increase on the lower-bound of the log likelihood of the data, thus improving the model, if trained properly. Once sufficiently many layers have been learned the deep architecture may be used as a generative model by reproducing the data when sampling down the model (an "ancestral pass") from the top level feature activations. Hinton reports that his models are effective feature extractors over high-dimensional, structured data.

Long short-term memory

Another technique particularly used for recurrent neural networks is the long short-term memory (LSTM) network of 1997 by Hochreiter & Schmidhuber. In 2009, deep multidimensional LSTM networks demonstrated the power of deep learning with many nonlinear layers, by winning three ICDAR 2009 competitions in connected handwriting recognition, without any prior knowledge about the three different languages to be learned.

Faster hardware

Hardware advances have meant that from 1991 to 2015, computer power (especially as delivered by