Connectors make Apache Airflow
Being able to connect to external sources and tools is what makes Apache Airflow a versatile tool and orchestrator. It also reenforces the need to complete all compute outside of Apache Airflow. Some key examples of connections that Apache Airflow can make are as follows:
- Connecting to an external database and extracting data for a data warehouse, data lake/lakehouse, or data mart
- Connecting to an API similar to our example and retrieving data for an ML model
- Connecting to a service provider with webhooks such as Microsoft Teams or Slack
Computing outside of Airflow
Running compute-heavy tasks outside of Airflow is a key way to ensure the orchestrator meets your team’s needs at scale. This is critical for performance as Airflow was designed to manage workflows, not to run heavy compute tasks. Running these compute tasks inside of Airflow can slow down workflows and make them prone to failure. In addition, Airflow costs...
