Creating custom operators

As seen in the previous recipe, Creating DAGs, it is nearly impossible to create a DAG without instantiating a task or, in other words, defining an operator. Operators are responsible for holding the logic required to process data in the pipeline.

We also know that Airflow already has predefined operators, allowing dozens of ways to ingest and process data. Now, it is time to put into practice how to create custom operators. Custom operators allow us to apply specific logic to a related project or data pipeline.

You will learn how to create a simple customized operator in this recipe. Although it is very basic, you will be able to apply the foundations of this technique to different scenarios.

In this recipe, we will create a custom operator to connect to and retrieve data from the HolidayAPI, the same as we saw previously, in Chapter 2.

Getting ready

Please, refer to the Getting ready section in the Configuring Airflow recipe for this recipe...