Designing our DAG
As we review the code, we notice a few important things for each of the notebooks for these models; they’re using identical code in a few sections to download and process data and for the “training” phase of the deep learning approach. After a little bit of time with a whiteboard and examining the code with your data scientist friend, you agree on the following design for your DAG.
Figure 8.1: The design of our DAG
Here are some things you noticed during the design phase that will impact implementation details:
- Since you don’t know exactly when data will become available for this process, it’s very hard to utilize a purely scheduled approach; instead, we’ll have to make sure our first step can determine whether the data has been changed before downloading and running all of these models.
- The notebooks your friend showed you had some repetitive code in them; you collapsed them into...
