Question

The Rectified Linear Unit operation on a single element x is defined as follows: ReLU(x) = max(0, x) What’s the benefit of using ReLU as activation function instead of using Sigmoid or Tanh activation functions?

Answer 1

• At first , the Rectified Linear Unit operation on a single element x defined by ReLU(X)=max(0,x).
• It means that if x is less than 0 (x<0) then ReLU(X) is equal to 0 and ,
• if x greater than or equal to 0 then ReLU(X) = x .
• The benefit of using ReLU is as follows ;
• The activation function ReLU is the one which is very simple to use.
• It is more efficient than the other two (Sigmoid and Tanh activation).
• Tanh means hyperbolic tangent function
• By using sigmoid activation function there is some disadvantages like killing of gradients.
• Also convergence of sigmoid is better.
• Therefore by using ReLu we can reduce the problems caused to gradients (i.e., the vanishing of gradient problem )
• Another benefit of ReLU is sparsity.
• ReLU not requires complex exponential calculations.
• The run time of ReLU is almost smaller.