Using Airflow as a Driving Service

Hopefully, by this point in your Airflow journey, the statement Airflow is a capable tool for the definition of orchestration workflows, is non-controversial. It should also be non-controversial to understand that, while powerful, it does require a more than cursory understanding of Python and Airflow internals to make it particularly valuable. At some point in your career, you will likely find a use case to abstract the authoring of workflows to a “less technical” group, allowing them to author and schedule workflows without having to have knowledge of Airflow or Python.

This chapter demonstrates a pattern in which Airflow is abstracted away from users. We do this to provide a simplified interface to Airflow that allows individuals to utilize Airflow to orchestrate and execute workloads without having to understand Airflow (or even know that it is responsible, as the backend service, for execution).

In this chapter, we are going...