Exploring RedisAI
We have seen the optimization we can get through TorchScript, but what will we do with the optimized binary? Yes, we can load it in the C++ world and make a Go server, and load it there, but that's still painful.
Redis Labs and Orobix brought us another solution called RedisAI. It is a highly optimized runtime built on top of LibTorch and can accept compiled TorchScript binaries for serving through the Redis protocol. For people who don't have prior experience with Redis, http://redis.io has good documentation and the introduction doc given there [3] should be a good start.
RedisAI comes with three options to configure three backends: PyTorch, TensorFlow, and ONNX Runtime. It doesn't stop there: RedisAI uses DLPack in the backend to enable tensors to pass through different frameworks without much conversion cost.
What does that even mean? Let's say you have a TensorFlow model that converts the human face to a 128-dimensional embedding (this...