The activation functions are the last piece of the neural network that we have not covered in depth yet. To review what we know so far, in a neural network, we start with an input, as we would with any machine-learning modeling exercise. This data consists of a dependent target variable that we would like to predict and any number of independent predictor variables that are to be used for this prediction task.
During the training process, the independent variables are weighted and combined in simulated neurons. A bias function is also applied during this step and this constant value is combined with the weighted independent variable values. At this point, an activation function evaluates an aggregation of the values and if it is above a set threshold limit, then the neuron fires and the signal is passed forward to additional hidden layers, if they...
