Various Optimizers

-

Optimizer

Optimers methods of algorithms are used to change the weights and learning rates of networks in order to reduce the losses.

Gradient descent

Gradient descent is one of the basic optimizers in deep learning. It is used in linear regression and classification problems. Gradient descent is a first-order optimization algorithm it is dependent on the first-order derivative of the loss function. It calculates the way Of altering the weights in order to reduce loss function and reach the minima.
θ=θ−α⋅∇J(θ)

Advantages

1. Easy understanding and computation

Disadvantages

1. Traps at local minima
2. Weights are changed after the gradient is found for the whole dataset.
3. More memory is required

Stochastic gradient descent

This is a variant of gradient descent this algorithm changes which more frequently. In this model both the weights and learning rates are change after the loss function for each training data is calculated.
θ=θ−α⋅∇J(θ;x(i);y(i))
where x(i) and y(i) are training data.

Advantages

1. Converges to a minimum very fast
2. Requires very less memory

Disadvantage

1. High variance in learning rate and weights
2. shoots even after converging to global Minima

Mini-batch gradient descent

This algorithm is more efficient than standard gradient descent and forecasting gradient descent. In this model, the training data set is divided into many batches and both the learning rate and weights are updated after every batch. 
θ=θ−α⋅∇J(θ; B(i))

where, B(i) is a batch

Advantages

1. Required medium amount of memory
2. It has less variance as frequently the weights and learning rates are changed.

so, choosing the optimal learning rate is very challenging and may trap at local minima.

Gradient descent with momentum

Momentum was invented to reduce the variance in stochastic gradient descent. This model will make the convergence soften and will move towards a relevant direction.

Advantages

1. Reduces the oscillations and high variants of learning rate and weights.
2. Converges very fast

Disadvantage

1. One more hyperparameter is added and that should be selected manually

RMSprop

The Root Means Square Propagation optimizer is similar to the gradient descent algorithm with momentum. The RMSprop optimizer restricts the oscillations in the vertical direction. Therefore, we can increase our learning rate and our algorithm could take larger steps in the horizontal direction converging faster. The difference between RMSprop and gradient descent is on how the gradients are calculated. The value of momentum is denoted by beta and is usually set to 0.9.

Advantages

1. Converges very fast

Adagrad

One of the disadvantages of all the optimizers explained is that the learning rate is constant for all parameters and for each cycle. This optimizer changes the learning rate. It changes the learning rate ‘η’ for each parameter and at every time step ‘t’. It’s a type second-order optimization algorithm. It works on the derivative of an error function.
η is a learning rate that is modified for given parameter θ(i) at a given time based on previous gradients calculated for given parameter θ(i). We store the sum of the squares of the gradients w.r.t. θ(i) up to time step t, while ϵ is a smoothing term that avoids division by zero (usually on the order of 1e−8). Interestingly, without the square root operation, the algorithm performs much worse.
It makes big updates for less frequent parameters and a small step for frequent parameters.

Advantages

1. Learning rate changes automatically after each cycle manual changes not required

Disadvantages

1. Computationally it is expensive as we have to calculate the second-order derivative
2. Decaying in learning rate.

Ada Delta

Ada delta is an extension of Adagrad. This model will rectify the disadvantage of Adagrad, as it tends to remove the decaying learning rate. Instead of accumulating all previously squared gradients, Adadelta limits the window of accumulated past gradients to some fixed size w. In this exponentially moving average is used rather than the sum of all the gradients.
We set γ to a similar value as the momentum term, around 0.9.

Advantages

1. Here the learning rate will not detail and training will not be stopped

Disadvantage

1. Computationally expensive

Adam

Adam (Adaptive Moment Estimation) works with momentums of first and second order. 
Adam takes into account the advantage of both Adagrad and RMS prop. The key idea of the Adam is that Instead of adapting the parameter learning rates based on the average first moment (the mean) as in RMSProp, Adam also makes use of the average of the second moments of the gradients (the uncentered variance).
M(t) and V(t) are values of the first moment which is the Mean and the second moment which is the uncentered variance of the gradients respectively.

Advantages

1. it is a very effective and fast method
2. It also rectifiers vanishing learning rates and also I variants

Disadvantage

1. It is computationally expensive

Conclusion

Above all stated optimizer algorithms, the best wellness Adam optimizer because if one wants to train the model effectively and with less time Adam is the one. Also in gradient descent mini-batch gradient descent is good.

Comments

Post a Comment

Popular posts from this blog

Deep Learning

-
Image
Deep Learning What is Deep Learning? Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks based learning from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. This is also similar to the Human Brain. Do you know the First Neuron created? Yes, Perceptron is the first neural network unit, that does some computations to detect features and business intelligence in input data. This was introduced by  Frank Rosenblatt in 1957. Types of Deep Learning? Artificial Neural Network Deep Brief Net Convolutional Neural Network Recurrent Neural Network Neural Network The first layer is the Input Layer, next is the Hidden layer and the final one is the Output layer. there can be any number of hidden layers in the network. How Neural Network works? Neural Network we create works similar to the human brain, we give features th
Read more

Loss & Cost Functions

-
Image
Loss Functions In neural networks, after a forward propagation, the loss function is calculated. The loss function is the result of the difference between actual output and the predicted output. Different loss functions will return different values for the same prediction, and thus have a considerable effect on model performance.\ Regression Loss Functions Mean Squared Error Loss Mean Squared Logarithmic Error Loss Mean Absolute Error Loss Binary Classification Loss Functions Binary Cross-Entropy Hinge Loss Squared Hinge Loss Multi-Class Classification Loss Functions Multi-Class Cross-Entropy Loss Sparse Multiclass Cross-Entropy Loss Kullback Leibler Divergence Loss Cost Functions Cost functions for Regression problems: Mean Error (ME) Mean Squared Error (MSE) Mean Absolute Error (MAE) Root Mean Squared Error (RMSE) Categorical Cross Entropy Cost Function. Binary Cross Entropy Cost Function. Mean Absolute Error This function returns the mean of absolute differences among predictions an
Read more

Recurrent Neural Network & LSTM

-
Image
Recurrent Neural Network RNN is a neural network, where the output of the previous step is fed as input to the current step . The reason why the previous output is required in cases likes to predict the next word in the sentence, hence remembrance of previous words. Hence RNN has loops in them, allowing information (words) to persist. Here, A is a network, A gets an input X  and leaves an output value h.  A loop allows information to be passed from one step of the network to the next. RNN is multiple copies of the same network, each passing the same information to the successor. Applications of RNN Chat Bots NLP Translator Sentence completer Stock price Predictor, etc The problem in  Simple RNN The problem is Long-term Dependencies . For example, consider a language model trying to predict the next word based on the previous word. If we are trying to predict the last word in "The clouds are in the sky," we don’t need any further context – it’s pretty obvious the next word i
Read more