Vanishing Gradient Problem

-
Vanishing Gradient Problem

What & How?

The problem is that during the backpropagation (from the final layer to the initial layer) for updating the weights in the large neural network with specific activation function, the gradient (derivative) value calculated for the initial (front) layers are very small, and the old weight and new weight will be approximately equal. This will make the network training slow. This is the vanishing gradient problem.



Why?

This problem is mainly because of some specific activation functions like Sigmoid and Tanh. The derivatives of these functions will range from (0,1). Therefore, the large value to the function will return a small value. Thus the derivative will be small.

This is not a problem for small neural networks, but a big problem for the dense networks. During the backpropagation, the gradients are being found from the final layer to the initial layer. By the chain rule, the derivative of each hidden layer is calculated.




For the n Hidden layers with sigmoid or tanh activation functions, the derivatives are multiplied. And, this result is multiplied with the learning rate. Thus the gradient of the Initial layers is exponentially decreased. So, the weights for these layers are not updated properly. So, the Old weight and New weight will approximately equal.


Solution?

  • ReLU Activation Function can be replaced with Sigmoid Function.
  • Reducing the number of layers.








Comments

Post a Comment

Popular posts from this blog

Sigmoid Activation Function

-
Image
Sigmoid or Logistic Activation Function Sigmoid Activation Function translates the output to the range (0;1). For small values (<-5), sigmoid returns a value close to zero, and for large values (>5) the result of the function gets close to 1. It is non Zero Centric. Execute  the Following Code Advantages: It is mostly used in the output layer for binary classification. But there are other activation function perform more effectively than Sigmoid Function Disadvantages: The exp( ) function is computationally expensive. The problem of vanishing gradients Not useful for the regression tasks as well.
Read more

ReLU Activation Function

-
Image
ReLU Activation  Function ReLU is the most commonly used activation function, because of its simplicity, not computationally expensive.  Advantages: It is cheap to compute. It is easy to optimize converges fast No vanishing gradient problem It is capable of outputting a true zero value allowing the activation of hidden layers in neural networks to contain one or more true zero values called Representational Sparsity Disadvantages: The downside for being zero for all negative values called dying ReLU. So if once a neuron gets negative it is unlikely for it to recover. This is called the “dying ReLU” problem If the learning rate is too high the weights may change to a value that causes the neuron to not get updated at any data point again. ReLU generally not used in RNN because they can have very large outputs so they might be expected to be far more likely to explode than units that have bounded values.
Read more

Deep Learning

-
Image
Deep Learning What is Deep Learning? Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks based learning from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. This is also similar to the Human Brain. Do you know the First Neuron created? Yes, Perceptron is the first neural network unit, that does some computations to detect features and business intelligence in input data. This was introduced by  Frank Rosenblatt in 1957. Types of Deep Learning? Artificial Neural Network Deep Brief Net Convolutional Neural Network Recurrent Neural Network Neural Network The first layer is the Input Layer, next is the Hidden layer and the final one is the Output layer. there can be any number of hidden layers in the network. How Neural Network works? Neural Network we create works similar to the human brain, we give feature...
Read more