Generative and discriminative models

In classification tasks, our goal is to learn the parameters of a model that optimally maps features of the explanatory variables to the response variable. All of the classifiers that we have previously discussed are discriminativemodels, which learn a decision boundary that is used to discriminate between classes. Probabilistic discriminative models, such as logistic regression, learn to estimate the conditional probability P(y|x); they learn to estimate which class is most likely given the input features. Non-probabilistic discriminative models, such as KNN, directly map features to classes.

Generative models do not directly learn a decision boundary. Instead, they model the joint probability distribution of the features and the classes, P(x, y). This is equivalent to modelling the probabilities of the classes and the probabilities of the features given the classes. That is, generative models model how the classes generate features. Bayes' theorem can...