RNNs actually have a long history and were first developed in the 1980s. The Hopfield network, as the first neural network with recurrent links, was invented by John Hopfield in Neurons with graded response have collective computational properties like those of two-state neurons (PNAS. 1984 May; 81(10): 3088-3092).
Inspired by the Hopfield network, the fully connected neural network—the Elman network—was introduced in Finding structure in time (Cognitive Science, 1990 March; 14(2): 179-211). The Elman network has one hidden layer and a set of context units connected to the hidden layer. At each time step, the context units keep track of the previous values of the hidden units.
In 1992, Schmidhuber discovered the vanishing gradient problem due to memorizing long-term dependencies. Five years later, the long short-term memory (LSTM...
