Migrating Airflow
In the data engineering profession, you’re likely going to need to migrate something every few years due to reasons such as new technology selections, infrastructure changes, or disaster recovery. No matter what the reason for carrying out a migration that goes poorly, they generally are deeply impactful to business processes and operations. Taking time to plan and de-risk associated activities is critical for a successful migration.
In this chapter, we’ll cover the following areas:
- Work management activities to plan and execute a pipeline migration
- Technical approaches to do your migration.
- Specific methods to migrate from one Airflow instance to another
By the end of this chapter, you will have an understanding of the activities required to plan for and execute a successful migration.
