Activation Functions in Neural Network

Types of Activation Functions

There are several types of activation functions in neural networks, which can be broadly categorized into three groups:

• Binary Step Function
• Linear Activation Function
• Non-linear Activation Function

Binary Step Function

The binary step function utilizes a threshold limit to determine whether a node is active or inactive based on its input. This activation function compares the input to the threshold limit value. If the input exceeds the threshold value, the node is activated and produces an output; otherwise, it remains inactive and its output is zero.

Linear Activation Function

A linear activation function, also known as an “identity function,” performs no calculations and simply passes the input value to the next layer unaltered. This type of activation function in neural networks has a linear output and cannot be restricted to a specific interval. As a result, linear regression models often utilize these types of activation functions.

Non-linear Activation Function

In neural networks, nonlinear activation functions are the most commonly used types. These functions enhance the generalizability and adaptability of the model to various types of data. Incorporating nonlinear activation functions in neural networks adds an additional step to the calculation of each layer, but it is a crucial step. Without activation functions, nodes in a neural network perform simple linear computations on their inputs based on weights and biases. As a result, combining two linear functions results in a linear function, making the number of hidden layers in the neural network irrelevant, and all layers perform the same function. Therefore, using nonlinear activation functions is vital for training the neural network.

Below are some examples of nonlinear activation functions commonly used in neural networks:

• Sigmoid activation function
• Softmax activation function
• Swish activation function
• Tanh or Hyperbolic Tangent activation function
• Rectifier activation function
• Leaky RELUs activation function

The figure below shows an overview of some important nonlinear activation functions.