Automating your code with a DAG

To turn this into a DAG, a few things need to be completed. These include setting up the DAG as originally designed as well as making use of some new tools that we have not covered yet around operators.

In most examples and code generation throughout this book, we will provide the code upfront and step through the information specific to the information provided throughout the chapter. The code for this can be found in the GitHub repository.

1. First, we begin by importing the required Python and Airflow libraries to build the DAG and create a variable called dag_owner to define who the author of the DAG is. We’ll use the same libraries as our Jupyter Notebook exploration, as well as necessary Airflow imports:

   import json
   import pathlib
   import airflow
   import requests
   import requests.exceptions as request_exceptions
   from datetime import date
   from airflow import DAG
   from airflow.operators.bash import BashOperator
   from airflow.operators.python...