Understanding RNNs

A common feature of all the neural networks seen so far is that they don't have a memory. Networks formed by either fully connected layers or convolutional layers process each input independently so that it is isolated from the other layers. However, in RNNs, "the past" is taken into account, and this is done using its previous output as the state; so, an RNN layer will have two inputs, one is which is the standard input of the current vector, and the other being the output of the previous vector, as seen in the following diagram:

Figure 5.2 – RNN loop unfolded

The RNN implements this memory feature with an internal loop over the entire sequence of elements. Let's explain it with some pseudocode, as follows:

state = 0
for input in input_sequence:
     output = f(input, state)
     state = output

There are several types of RNN architectures with much more...