What is so exciting about recurrent neural networks?

Coming from a mathematics background, in my rather hectic career I have seen many different trends, particularly during the last few years, which all sound very similar to me: "you have a problem? wavelets can save you!", "finite elements are the solution to everything", and similar over-enthusiastic claims. 

Of course, each tool has its time and place and, more importantly, an application domain where it excels. I find recurrent neural networks quite interesting for the many features they can achieve:

• Produce consistent markup text (opening and closing tags, recognizing timestamp-like data)
• Write Wikipedia articles with references, and create URLs from non-existing addresses, by learning what a URL should look like
• Create credible-looking scientific papers from LaTeX

All these amazing features are...