Distributed training

One of the strengths of TF 2.0 is to be able to train and inference your model in a distributed manner on multiple GPUs and TPUs without writing a lot of code. This is simplified using the distribution strategy API, tf.distribute.Strategy(...), which is readily available for use. The fit() API section, which explains tf.keras.Model.fit(...), showed how this function was used to train a model. In this section, we will show how to train tf.keras-based models across multiple GPUs and TPUs using a distribution strategy. It's worth noting that tf.distribute.Strategy(...) is available with high-level APIs such as tf.keras and tf.estimator, along with having support for custom training loops as well or for any computation in general. Also, the distribution strategy described here is supported for eagerly executed programs, such as models written using TF 2.0...