Stateless Model Serving

In this chapter, we will talk about stateless model serving, the first pattern-based on serving philosophies. We will first talk about stateless and stateful functions to give you an introduction to these concepts. We will see that stateful functions depend on states within the model and also on the states from the previous calls. Due to these strongly coupled dependencies on states, it is difficult to scale the serving.

When serving is desired, the model is served in a stateless manner so that the model does not depend on previous calls, can be scaled easily, and the output is consistent. However, some machine learning models are by default stateful, and we can attempt to reduce the variance of the server model by using some tricks such as specifying random seeds and using hyperparameters that reduce the variance of the model due to states.

This chapter will give an overview of stateful and stateless functions along with examples. Then we will discuss...