Complexities in the ML process

Each iteration through the process is an experiment to see whether the changes that were made in a previous part of the process will yield a better result or a more optimized ML model. It is this process of iteration that makes the ML workflow hard and difficult to automate. The goal of each iteration or experiment is to improve the model's overall predictive capabilities. During each iteration, we fine-tune the parameters, discover new variables, and verify that these changes improve the overall accuracy of the model's prediction. Each experiment also provides further insight into where we are in the overall process and what the next steps might be. In essence, having to potentially go back and tweak a previous step or even go back to the very beginning of the process and start with a different set of data, parameters, or even a different ML model altogether is a manual process. But even unsuccessful experiments have value since they allow us to learn from our mistakes and hopefully steer us toward a successful outcome.

Note

Tolerating failures and not letting them derail the overall ML process is a key factor in any successful ML strategy.

So, if the overall process is complicated and executing the methodology yields failures, this will hopefully lead to a more successful outcome that will impact the overall ML strategy. It becomes noticeably clear why automating the entire process is challenging but necessary, as it now becomes a crucial part of the overall success criteria of any ML project.

Now that we have a good idea of what makes the ML process difficult, let's explore these challenges further by covering a practical example.