Working with Hugging Face models

The Hugging Face Transformers library has brought together multiple pre-trained transformer models to solve relevant NLP tasks such as text classification, text generation, and information extraction. These models include BERT, RoBERTa, GPT, GPT-2, and other state-of-the-art transformer models. What is the advantage of using pre-trained models? With pre-trained models and transfer learning, we can come up with accurate models in a shorter period of time. That's because we can start with a good set of weights from models that have been trained to solve a similar set of problems.

In this recipe, we will start with the pre-trained DistilBERT model and use the HuggingFace estimator class from the SageMaker Python SDK, along with a custom Python script file, to fine-tune our DistilBERT model. We will use the synthetic text data from the Generating a synthetic dataset for text classification problems recipe of Chapter 8, Solving NLP, Image Classification...