Model assessment, evaluation, and comparisons
The key ideas discussed here are:
- How to assess or estimate the performance of the classifier on unseen datasets that it will be predicting on future unseen datasets.
- What are the metrics that we should use to assess the performance of the model?
- How do we compare algorithms if we have to choose between them?
Model assessment
In order to train the model(s), tune the model parameters, select the models, and finally estimate the predictive behavior of models on unseen data, we need many datasets. We cannot train the model on one set of data and estimate its behavior on the same set of data, as it will have a clear optimistic bias and estimations will be unlikely to match the behavior in the unseen data. So at a minimum, there is a need to partition data available into training sets and testing sets. Also, we need to tune the parameters of the model and test the effect of the tuning on a separate dataset before we perform testing on the test set. The...
