Recurrent neural networks

RNN is a type of neural network, which can process sequential data with variable length. Examples of such data include the words of a sentence or the price of a stock in various moments of time. By using the word sequential, we imply that the elements of the sequence are related to each other and their order matters. For example, if we take a book and shuffle randomly all the words in it, the text will loose it's meaning, even though we'll still know the individual words.

RNNs get their name because they apply the same function over a sequence recurrently. We can define an RNN as a recurrence relation:

Here, f is a differentiable function, s_t is a vector of values called internal network state (at step t), and x_t is the network input at step t. Unlike regular networks, where the state only depends on the current input (and network weights)...