Generating Text with RNNs and GPT-2
When your mobile phone completes a word as you type a message or when Gmail suggests a short reply or completes a sentence as you reply to an email, a text generation model is working in the background. The Transformer architecture forms the basis of state-of-the-art text generation models. BERT, as explained in the previous chapter, uses only the encoder part of the Transformer architecture.
However, BERT, being bi-directional, is not suitable for the generation of text. A left-to-right (or right-to-left, depending on the language) language model built on the decoder part of the Transformer architecture is the foundation of text generation models today.
Text can be generated a character at a time or with words and sentences together. Both of these approaches are shown in this chapter. Specifically, we will cover the following topics:
- Generating text with:
- Character-based RNNs for generating news headlines...
