Model complexity measures for model selection
Practical model complexity measures tend not to measure model complexity directly. Instead, they measure some sort of trade-off – for example, how much information has been lost by approximating the patterns present in a dataset by using a particular model form, or what evidence a dataset provides for a model form of this level of complexity. These practical metrics don’t directly measure model complexity, but they take it into account.
Selecting between classes of models
In the preceding paragraph, we referred to model form. But what do we mean by model form? We mean the mathematical form of the equation that defines a model. So, two models that differ only in their parameter values but otherwise have the same form of mathematical equation have the same model form (e.g., two linear models that use the same features).
A model form represents a whole class of models. Let’s go back to our polynomial model example...
