Comparing XGBoost to gradient boosting and random forest models

In this section, you will build a gradient-boosted tree model and a random forest model with scikit-learn and compare them to XGBoost. Gradient boosting and random forest are types of ensemble tree-based models. Recall from Chapter 3 that the authors of XGBoost made several improvements to speed up execution, but many of the changes they made were for scalability so that the algorithm could handle large datasets effectively. What does the regular gradient boosting tree model do on this small example dataset? In this section, you will build a gradient boosting model and a random forest model using scikit-learn. scikit-learn is consistent in how models are called and fit, so the steps here will be similar to what you have just completed for CART. Let’s get started:

1. Fit a regression model using gradient-boosted trees: Import GradientBoostingRegressor from sklearn.ensemble, which contains the ensemble models...