GRU

-

Gated Recurrent Unit

GRU is considered to solve the vanishing gradient problem, it is a variation of LSTM. It is an improved version of RNN. GRU uses 2 gates i.e, Update and Reset Gates, this makes it special. Basically, these are two vectors that decide what information should be passed to the output. The special thing about them is that they can be trained to keep information from long ago, without washing it through time or remove information which is irrelevant to the prediction.

Update Gate

This is the single GRU cell. Here we have 2 inputs and 1 output. x(t) is the new information passed and h(t-1) is the previous information, Update gate will decide which information is to be updated or forgotten. That is really powerful because the model can decide to copy all the information from the past and eliminate the risk of vanishing gradient problem. 
So, the sigmoid operation is done with both current input and previous information. In sigmoid operation, In the sigmoid operation, the network decides what information is relevant to continue between the previous output and the new input assigning values from 0 to 1 where 0 goes for the information we want to forget and 1 for the information we want to keep. This result is then multiplied to the previous output to continue for the next step.

Reset Gate

Essentially, this gate is used from the model to decide how much of the past information to forget. The path preceding the first one comes with the combination of the previous output and the current input, here the networks decide ones again what is relevant by sigmoid. The first result from the sigmoid is multiplied with -1 and multiplied with the clean output coming directly from the previous cell. The second result from sigmoid is then multiplied with the result of the activation function tanh applied to the result of the first step gate multiplication.
This last part is carrying the importance of the new information that enters in this particular cell. The idea of separate the result into two paths, one multiplied by 1 and the other by -1 is to reinforce the separation of concern between forget the majority of the previous output or keep it, taking into account that while a state is closing to 0 saying to forget what is coming through the other is close to 1 saying to save what is coming for this side.

This is also similar to LSTM but with some variation version of LSTM RNN.


Comments

Post a Comment

Popular posts from this blog

Deep Learning

-
Image
Deep Learning What is Deep Learning? Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks based learning from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. This is also similar to the Human Brain. Do you know the First Neuron created? Yes, Perceptron is the first neural network unit, that does some computations to detect features and business intelligence in input data. This was introduced by  Frank Rosenblatt in 1957. Types of Deep Learning? Artificial Neural Network Deep Brief Net Convolutional Neural Network Recurrent Neural Network Neural Network The first layer is the Input Layer, next is the Hidden layer and the final one is the Output layer. there can be any number of hidden layers in the network. How Neural Network works? Neural Network we create works similar to the human brain, we give features th
Read more

Loss & Cost Functions

-
Image
Loss Functions In neural networks, after a forward propagation, the loss function is calculated. The loss function is the result of the difference between actual output and the predicted output. Different loss functions will return different values for the same prediction, and thus have a considerable effect on model performance.\ Regression Loss Functions Mean Squared Error Loss Mean Squared Logarithmic Error Loss Mean Absolute Error Loss Binary Classification Loss Functions Binary Cross-Entropy Hinge Loss Squared Hinge Loss Multi-Class Classification Loss Functions Multi-Class Cross-Entropy Loss Sparse Multiclass Cross-Entropy Loss Kullback Leibler Divergence Loss Cost Functions Cost functions for Regression problems: Mean Error (ME) Mean Squared Error (MSE) Mean Absolute Error (MAE) Root Mean Squared Error (RMSE) Categorical Cross Entropy Cost Function. Binary Cross Entropy Cost Function. Mean Absolute Error This function returns the mean of absolute differences among predictions an
Read more

Recurrent Neural Network & LSTM

-
Image
Recurrent Neural Network RNN is a neural network, where the output of the previous step is fed as input to the current step . The reason why the previous output is required in cases likes to predict the next word in the sentence, hence remembrance of previous words. Hence RNN has loops in them, allowing information (words) to persist. Here, A is a network, A gets an input X  and leaves an output value h.  A loop allows information to be passed from one step of the network to the next. RNN is multiple copies of the same network, each passing the same information to the successor. Applications of RNN Chat Bots NLP Translator Sentence completer Stock price Predictor, etc The problem in  Simple RNN The problem is Long-term Dependencies . For example, consider a language model trying to predict the next word based on the previous word. If we are trying to predict the last word in "The clouds are in the sky," we don’t need any further context – it’s pretty obvious the next word i
Read more