Long short-term memory networks

Long short-term memory networks (LSTMS), are a special type of RNN capable of learning long-term dependencies. While standard RNNs can remember previous states to some extent, they did this on a fairly basic level by updating a hidden state on each time step. This enabled the network to remember short-term dependencies. The hidden state, being a function of previous states, retains information about these previous states. However, the more time steps there are between the current state and a previous state, it diminishes the effect that this earlier state will have on the current state. Far less information is retained on a state that is say 10 time steps before the time step immediately preceding the current step. This is despite that fact that earlier time steps may contain important information with direct relevance to a particular problem or...