Implementing champion/challenger model management

In most real-world predictive analytics applications, models must change over time. This need for change is often due to evolving customer behavior, new offerings/promotions, and/or changes in data availability. Regardless of the reason necessitating change, it's often advantageous to automate the process of building updated models. Frequency of the model refresh process depends on the nature of the business. In some rapidly changing businesses, the refresh process is sub-hourly and automation is an absolute necessity.

With the champion/challenger technique, the currently deployed model is called the champion model. New models built by training on the latest data are called the challenger models. Challenger models can replace the champion model if the challenger is more effective than the champion model. Model effectiveness can be defined many ways including, but not limited to, mean absolution percent error (MAPE) for continuous targets and...