Using forward and backward feature selection

Forward and backward feature selection, as their names suggest, select features by adding them one by one – or subtracting them for backward selection – and assessing the impact on model performance after each iteration. Since both methods assess that performance based on a given algorithm, they are considered wrapper selection methods.

Wrapper feature selection methods have two advantages over the filter methods we have explored so far. First, they evaluate the importance of features as other features are included. Second, since features are evaluated based on their contribution to the performance of a specific algorithm, we get a better sense of which features will ultimately matter. For example, satmath seemed to be an important feature based on our results from the previous section. But it is possible that satmath is only important when we use a particular model, say linear regression, and not an alternative such as...