The output from the neuron is computed as shown in Figure 3, and passed through an activation function that introduces non-linearity to the output. This f is called an activation function. The main purposes of the activation functions are to:
- Introduce nonlinearity into the output of a neuron. This is important because most real-world data is nonlinear and we want neurons to learn these nonlinear representations.
- Squash the output to be in a specific range.
Every activation function (or nonlinearity) takes a single number and performs a certain fixed mathematical operation on it. There are several activation functions you may encounter in practice.
So, we are going to briefly cover the most common activation functions.
