Building our data pipeline application

We now have our ETL app. Configuring it is more complex than the schema app but basically works the same.

Here, we define our REST APIs; each API has a name that matches the table name and a URL:

etl-jobs/etl_jobs/configuration/api_raw_data/apis.py
"""
fill in
"""

We have a tiny import section but let’s start here first:

import abc
from dataclasses import dataclass

Here we follow the same base configuration pattern. First, we create a base class and then we inherit that class, adding information. This pattern allows us to create uniformity and control the look of each class:

class restConfig(abc.ABC):
    """
    fill in
    """
    name = ""
    url = ""

We will now inherit from the base class and add information to our configuration:

...