Chapter 4 - Generating Song Lyrics Using an RNN

1. A normal feedforward neural network predicts output only based on the current input, but an RNN predicts output based on the current input and also the previous hidden state, which acts as a memory and stores the contextual information (input) that the network has seen so far.

2. The hidden state, , at a time step, , can be computed as follows:

   

   In other words, this is hidden state at a time step, t = tanh([input to hidden layer weight x input] + [hidden to hidden layer weight x previous hidden state]).

3. RNNs are widely applied for use cases that involve sequential data, such as time series, text, audio, speech, video, weather, and much more. They have been greatly used in various natural language processing (NLP) tasks, such as language translation, sentiment analysis, text generation, and so on.
4. While backpropagating the RNN,...