Activation Functions

-

Activation Functions

What is the Activation Function?

  • Activation Functions are mathematical equations that determine the output of the neural network. It is applied to every neuron and determines when should each neuron be activated and deactivated. 

  • For example, if I pour hot water on your left hand, you suddenly move away from your left hand but not your right hand. This is because only your left-hand neuron is activated and rest of the neurons are deactivated.

  • These functions also normalize the output of each neuron to range between 0 and 1 or -1 and 1.

Why Activation Function?

  • We provide a set of training samples to a computer for training and the computer will adjust the weight and bias value according to minimize the output. 
  • While a computer is training some image samples, we would like neural networks to according change the weights and bias values so, fewer images are recognized wrongly. But our neural network will not change the weights and bias value little by little, it jumps here and there, no continuous change.
  • So, to make the changes continuously we need a relevant continuous function to compute derivative called Activation Function.

Types of Activation Functions:

  1. Sigmoid or Logistic Activation Function
  2. tanh Activation Function
  3. ReLU Activation Function
  4. Maxout Activation
  5. SoftMax Activation
  6. SoftPlus Activation
  7. SoftSign Activation
  8. ELU Activation
  9. Exponential Activation
  10. Selu Activation

Comments

Post a Comment

Popular posts from this blog

Sigmoid Activation Function

-
Image
Sigmoid or Logistic Activation Function Sigmoid Activation Function translates the output to the range (0;1). For small values (<-5), sigmoid returns a value close to zero, and for large values (>5) the result of the function gets close to 1. It is non Zero Centric. Execute  the Following Code Advantages: It is mostly used in the output layer for binary classification. But there are other activation function perform more effectively than Sigmoid Function Disadvantages: The exp( ) function is computationally expensive. The problem of vanishing gradients Not useful for the regression tasks as well.
Read more

Various Optimizers

-
Image
Optimizer Optimers methods of algorithms are used to change the weights and learning rates of networks in order to reduce the losses. Gradient descent Gradient descent is one of the basic optimizers in deep learning. It is used in linear regression and classification problems. Gradient descent is a first-order optimization algorithm it is dependent on the first-order derivative of the loss function. It calculates the way Of altering the weights in order to reduce loss function and reach the minima. θ=θ−α⋅∇J(θ) Advantages 1. Easy understanding and computation Disadvantages 1. Traps at local minima 2. Weights are changed after the gradient is found for the whole dataset. 3. More memory is required Stochastic gradient descent This is a variant of gradient descent this algorithm changes which more frequently. In this model both the weights and learning rates are change after the loss function for each training data is calculated. θ=θ−α⋅∇J(θ;x(i);y(i)) where x(i) and y(i) are training data....
Read more

Convolutional Neural Network

-
Image
Convolutional Neural Network      CNN/ConvNet is a Deep Learning algorithm, that deals with images, it takes an input image and assigns a learnable weight and bias values, and learns to differentiate with other images . Unlike other neural networks, the preprocessing required in ConvNet is smaller. In primitive methods filters are handed engineered, with enough training and ConvNets have the ability to learn them easily. The Architecture of ConvNet is analogous to the human system of the visual cortex (connectivity pattern of neurons in the Human Brain).           Individual Neurons respond only to stimuli only the restricted region of the visual field. This is called the receptive field, and the Collection of such fields is called a visual area. What is an Image? An Image is nothing but a matrix of pixel values.  3x3 image matrix is converted into 9x1 vector.  A ConvNet is able to successfully capture the Spatial and Temporal depen...
Read more