Ordinary Least Squares as a Classifier

We covered ordinary least squares (OLS) as linear regression in the context of predicting continuous variable output in the previous chapter, but it can also be used to predict the class that a set of data is a member of. OLS-based classifiers are not as powerful as other types of classifiers that we will cover in this chapter, but they are particularly useful in understanding the process of classification. To recap, an OLS-based classifier is a non-probabilistic, linear binary classifier. It is non-probabilistic because it does not generate any confidence over the prediction such as, for example, logistic regression. It is a linear classifier as it has a linear relationship with respect to its parameters/coefficient.

Now, let's say we had a fictional dataset containing two separate groups, Xs and Os, as shown in Figure 5.1. We could construct a linear classifier by first using OLS linear regression to fit the equation of a straight line...