Exploding Gradient Problem

-

Exploding Gradient Problem

What & How?

The problem is that during the backpropagation traversing from the final layer to the initial layer for updating the weights in the large neural network the n derivative values are multiplied. When the derivative value is large then the gradient value exponentially increases and makes the model more unstable. This is the Exploding Gradient Problem. During the backpropagation, the large change in the weights makes the model very unstable.

Solutions?

  • Weight Initialization.
  • Reducing the number of layers.
  • Gradient Clipping

Weight Initialization?

A careful initialization of weights should be assigned during backpropagation. This can be achieved using Random initialization.

Reduce the number of layers?

By reducing the number of hidden layers, the problem of exploding and vanishing gradient problem can be solved.

Gradient Clipping?

Gradient Clipping is a technique that tackles the problem of exploding gradient problem. The idea is very simple, that if the gradient value is very large, this rescales to value to small, if ||g|| > c, then
where,
        g = gradient
        c = hyperparameter
        ||g|| = norm of gradient
since g/||g|| is a unit vector, after rescaling the new g will have norm c. Note that if ||g|| < c, then we don’t need to do anything.



Comments

Post a Comment

Popular posts from this blog

Deep Learning

-
Image
Deep Learning What is Deep Learning? Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks based learning from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. This is also similar to the Human Brain. Do you know the First Neuron created? Yes, Perceptron is the first neural network unit, that does some computations to detect features and business intelligence in input data. This was introduced by  Frank Rosenblatt in 1957. Types of Deep Learning? Artificial Neural Network Deep Brief Net Convolutional Neural Network Recurrent Neural Network Neural Network The first layer is the Input Layer, next is the Hidden layer and the final one is the Output layer. there can be any number of hidden layers in the network. How Neural Network works? Neural Network we create works similar to the human brain, we give features th
Read more

Loss & Cost Functions

-
Image
Loss Functions In neural networks, after a forward propagation, the loss function is calculated. The loss function is the result of the difference between actual output and the predicted output. Different loss functions will return different values for the same prediction, and thus have a considerable effect on model performance.\ Regression Loss Functions Mean Squared Error Loss Mean Squared Logarithmic Error Loss Mean Absolute Error Loss Binary Classification Loss Functions Binary Cross-Entropy Hinge Loss Squared Hinge Loss Multi-Class Classification Loss Functions Multi-Class Cross-Entropy Loss Sparse Multiclass Cross-Entropy Loss Kullback Leibler Divergence Loss Cost Functions Cost functions for Regression problems: Mean Error (ME) Mean Squared Error (MSE) Mean Absolute Error (MAE) Root Mean Squared Error (RMSE) Categorical Cross Entropy Cost Function. Binary Cross Entropy Cost Function. Mean Absolute Error This function returns the mean of absolute differences among predictions an
Read more

Recurrent Neural Network & LSTM

-
Image
Recurrent Neural Network RNN is a neural network, where the output of the previous step is fed as input to the current step . The reason why the previous output is required in cases likes to predict the next word in the sentence, hence remembrance of previous words. Hence RNN has loops in them, allowing information (words) to persist. Here, A is a network, A gets an input X  and leaves an output value h.  A loop allows information to be passed from one step of the network to the next. RNN is multiple copies of the same network, each passing the same information to the successor. Applications of RNN Chat Bots NLP Translator Sentence completer Stock price Predictor, etc The problem in  Simple RNN The problem is Long-term Dependencies . For example, consider a language model trying to predict the next word based on the previous word. If we are trying to predict the last word in "The clouds are in the sky," we don’t need any further context – it’s pretty obvious the next word i
Read more