The bias/variance trade-off

In this section, we're going to continue our discussion of error due to bias, and introduce a new source of error called variance. We will begin by clarifying what we mean by error terms and then dissect various sources of modeling errors.

Error terms

One of the central topics of model building is reducing error. However, there are several types of errors, two of which we have control over to some extent. These are called bias and variance. There is a trade-off in the ability for a model to minimize either bias or variance, and this is called the bias-variance trade-off or the bias-variance dilemma.

Some models do well at controlling both to an extent. However, this is a dilemma that, for the most part, is always going to be present in your modeling considerations.

Error due to bias

High bias can also be called underfitting or over-generalization. High bias generally leads to an inflexible model that misses the true relationship between features in the target function...