Installing ML libraries on Databricks

Databricks is a unified big data and analytics platform. It is great for training ML models and working with the kind of large-scale data that is often found in IoT. There are extensions such as Delta Lake that allow researchers the ability to view data as it existed at certain periods of time so that they can do analysis when models drift. There are also tools such as MLflow that allow the data scientist to compare multiple models against each other. In this recipe, we are going to install various ML packages such as TensorFlow, PyTorch, and GraphFrames on Databricks. Most ML packages can be installed via PyPI. The format used to install TensorFlow, for example, will work on various ML frameworks such as OpenAI Gym, Sonnet, Keras, and MXNet. Some tools are available in Databricks that are not available in Python. For those, we use the pattern explored by GraphX and GraphFrame where packages are installed through Java extensions.