Creating validation sets

Throughout the book, we mentioned many times that we need to experiment with multiple configurations of the models to find the optimal one. The most typical pipeline is adjusting the hyperparameters and the topology of deep learning architecture, training on a set of samples, and testing on another set. For that reason, machine learning is a highly iterative process. This strategy engenders a particular risk, however. Evaluating different model configurations with a given test set over multiple rounds leads to a model tuned to work well with the specific set. As the number of epochs increases, we implicitly fit the model to the peculiarities of the test set and consequently get a too-optimistic performance in the end.

We need a way to validate our model performance during training while leaving the test set for the final evaluation. This role is undertaken by the validation set that helps us tune the model’s hyperparameters and configurations accordingly...