Training versus testing error

Now that we have presented three very useful classifiers, it is time for us to evaluate their accuracy on the testing set; in the training set, the three models appear to give us about the same accuracy of about 80%. However, before calculating testing accuracy, recall what we said in the previous chapter about the need for a reference point to know if this 80% is good or bad. Back in the previous chapter, we answered a version of this question—in the absence of any information about the customer, what would be our best guess for his payment status next month? In this case, we have only two choices: pay or default, and since most of the clients in our sample paid, in the absence of any information our best guess would be to always predict pay. This simple strategy (always predict pay) will be in this case called the null model, the model without...