Examining the top solution ideas from Michael Jahrer

Michael Jahrer (https://www.kaggle.com/mjahrer, competition Grandmaster and one of the winners of the Netflix Prize in the team “BellKor’s Pragmatic Chaos”) led the public leaderboard for a long time and by a fair margin during the competition and was declared the winner when the private solutions were finally disclosed.

Shortly after, in the discussion forum, he published a short summary of his solution that has become a reference for many Kagglers because of his smart usage of denoising autoencoders and neural networks (https://www.kaggle.com/competitions/porto-seguro-safe-driver-prediction/discussion/44629). Although Michael hasn’t accompanied his post with any Python code regarding his solution (he described his coding work as an “old-school” and “low-level” one, being directly written in C++/CUDA with no Python), his writing is quite rich in references to what models he has used as well as their hyperparameters and architectures.

First, Michael explains that his solution is composed of a blend of six models (one LightGBM model and five neural networks). Moreover, since no advantage could be gained by weighting the contributions of each model to the blend (as well as doing linear and non-linear stacking), likely because of overfitting, he states that he resorted to just a blend of models (where all the models had equal weight) that have been built from different seeds.

Such insight makes the task much easier for us to replicate his approach, also because he also because he mentions that just having blended the LightGBM’s results with one from the neural networks he built would have been enough to guarantee first place in the competition.

This insight will limit our exercise work to two good single models instead of a host of them. In addition, he mentioned having done little data processing, besides dropping some columns and one-hot encoding categorical features.