Activation Functions

-

Activation Functions

What is the Activation Function?

  • Activation Functions are mathematical equations that determine the output of the neural network. It is applied to every neuron and determines when should each neuron be activated and deactivated. 

  • For example, if I pour hot water on your left hand, you suddenly move away from your left hand but not your right hand. This is because only your left-hand neuron is activated and rest of the neurons are deactivated.

  • These functions also normalize the output of each neuron to range between 0 and 1 or -1 and 1.

Why Activation Function?

  • We provide a set of training samples to a computer for training and the computer will adjust the weight and bias value according to minimize the output. 
  • While a computer is training some image samples, we would like neural networks to according change the weights and bias values so, fewer images are recognized wrongly. But our neural network will not change the weights and bias value little by little, it jumps here and there, no continuous change.
  • So, to make the changes continuously we need a relevant continuous function to compute derivative called Activation Function.

Types of Activation Functions:

  1. Sigmoid or Logistic Activation Function
  2. tanh Activation Function
  3. ReLU Activation Function
  4. Maxout Activation
  5. SoftMax Activation
  6. SoftPlus Activation
  7. SoftSign Activation
  8. ELU Activation
  9. Exponential Activation
  10. Selu Activation

Comments

Post a Comment

Popular posts from this blog

Deep Learning

-
Image
Deep Learning What is Deep Learning? Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks based learning from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. This is also similar to the Human Brain. Do you know the First Neuron created? Yes, Perceptron is the first neural network unit, that does some computations to detect features and business intelligence in input data. This was introduced by  Frank Rosenblatt in 1957. Types of Deep Learning? Artificial Neural Network Deep Brief Net Convolutional Neural Network Recurrent Neural Network Neural Network The first layer is the Input Layer, next is the Hidden layer and the final one is the Output layer. there can be any number of hidden layers in the network. How Neural Network works? Neural Network we create works similar to the human brain, we give features th
Read more

Loss & Cost Functions

-
Image
Loss Functions In neural networks, after a forward propagation, the loss function is calculated. The loss function is the result of the difference between actual output and the predicted output. Different loss functions will return different values for the same prediction, and thus have a considerable effect on model performance.\ Regression Loss Functions Mean Squared Error Loss Mean Squared Logarithmic Error Loss Mean Absolute Error Loss Binary Classification Loss Functions Binary Cross-Entropy Hinge Loss Squared Hinge Loss Multi-Class Classification Loss Functions Multi-Class Cross-Entropy Loss Sparse Multiclass Cross-Entropy Loss Kullback Leibler Divergence Loss Cost Functions Cost functions for Regression problems: Mean Error (ME) Mean Squared Error (MSE) Mean Absolute Error (MAE) Root Mean Squared Error (RMSE) Categorical Cross Entropy Cost Function. Binary Cross Entropy Cost Function. Mean Absolute Error This function returns the mean of absolute differences among predictions an
Read more

Recurrent Neural Network & LSTM

-
Image
Recurrent Neural Network RNN is a neural network, where the output of the previous step is fed as input to the current step . The reason why the previous output is required in cases likes to predict the next word in the sentence, hence remembrance of previous words. Hence RNN has loops in them, allowing information (words) to persist. Here, A is a network, A gets an input X  and leaves an output value h.  A loop allows information to be passed from one step of the network to the next. RNN is multiple copies of the same network, each passing the same information to the successor. Applications of RNN Chat Bots NLP Translator Sentence completer Stock price Predictor, etc The problem in  Simple RNN The problem is Long-term Dependencies . For example, consider a language model trying to predict the next word based on the previous word. If we are trying to predict the last word in "The clouds are in the sky," we don’t need any further context – it’s pretty obvious the next word i
Read more