Scaling up training

Scaling up training requires us to speed up the training process for large amounts of data and utilize GPUs and TPUs better. In this section, we will cover some of the tips on how to efficiently use provisions in PyTorch Lightning to accomplish this.

Speeding up model training using a number of workers

How can the PyTorch Lightning framework help speed up model training? One useful parameter to know is num_workers, which comes from PyTorch, and PyTorch Lightning builds on top of it by giving advice about the number of workers.

Solution

The PyTorch Lightning framework offers a number of provisions for speeding up model training, such as the following:

• You can set a non-zero value for the num_workers argument to speed up model training. The following code snippet provides an example of this:

  import torch.utils.data as data
  ...
  dataloader = data.DataLoader(num_workers=4, ...)

The optimal num_workers value depends on the batch size and configuration...