Connecting to External Sources
One of Apache Airflow’s most powerful tools and functions is its ability to connect to external sources and hooks. This allows powerful workflow management for data engineers to connect to external sources such as databases, APIs, and cloud storage providers. It also provides the ability to call on other tools within the data engineering ecosystem such as data transforming tools, such as dbt, and machine learning (ML) tools, such as TensorFlow.
At its most basic level, connectors can be used to call upon secrets and perform notification functions for your team. In this chapter, we will look at the basics of setting up a connector within the Apache AI user interface (UI), authoring some additional lines of code to alert the team of a DAG failure, and reviewing best practices for storing secrets.
In this chapter, we are going to cover the following main topics:
- Adding connectors to the Apache Airflow UI
- Expanding on previous DAG...
