Introducing keyed prediction
When we make batch predictions of multiple instances at the same time, we split the prediction input and ask multiple servers or multiple instances of the same model to make the prediction in parallel. I hope you have already asked yourself, if predictions are made concurrently, then how can we ensure an order? We know that ordering is difficult in concurrent programming and the order of predictions may be violated. This creates problems on the client side. Clients may be misguided when mapping predictions to features. As multiple servers running in parallel might give the responses in different orders, it is possible that the input instances passed will be returned in a different order.
Figure 5.1 – Having parallel model servers can cause a loss of order during prediction
For example, let’s consider the scenario shown in Figure 5.1. Here, we have two models in two servers, Model Server 1 and Model Server 2,...
