Treating imbalanced datasets

The pending issue from the beginning of the chapter concerns the preliminary observation that the dataset is imbalanced. Specifically, the class distribution has a severe skew, as the offensive tweets prevail in the corpus. Training machine-learning models without mitigating this concern engenders the risk of having a strong bias toward the majority class. A possible strategy to address this problem is to perform random oversampling by randomly duplicating examples in the minority class. Conversely, we can randomly delete examples in the majority class using random undersampling. In both cases, applying re-sampling strategies leads to more balanced data distributions.

In this section, we attack the problem differently and use class weighting. Based on the number of instances in each class, we calculate weights that the model can use to pay more attention to examples from the underrepresented classes:

# Calculate the number of instances per class....