Understanding Recurrent Neural Networks

In this section, we will discuss what an RNN is by starting with a gentle introduction, and then move on to more in-depth technical details. We mentioned earlier that RNNs maintain a state variable which evolves over time as the RNN is seeing more data, thus giving the power to model sequential data. In particular, this state variable is updated over time by a set of recurrent connections. Existence of recurrent connections is the main structural difference between an RNN and a feed-forward network. The recurrent connections can be understood as links between a series of memory RNN learned in the past, connecting to the current state variable of the RNN. In other words, the recurrent connections update the current state variable with respect to the past memory the RNN has, enabling the RNN to make a prediction based on the current input as well as the previous inputs.

In the upcoming section, we will discuss the following things. First, we will discuss...