Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Apache Airflow Best Practices

You're reading from   Apache Airflow Best Practices A practical guide to orchestrating data workflow with Apache Airflow

Arrow left icon
Product type Paperback
Published in Oct 2024
Publisher Packt
ISBN-13 9781805123750
Length 188 pages
Edition 1st Edition
Languages
Arrow right icon
Authors (3):
Arrow left icon
Dylan Storey Dylan Storey
Author Profile Icon Dylan Storey
Dylan Storey
Dylan Intorf Dylan Intorf
Author Profile Icon Dylan Intorf
Dylan Intorf
Kendrick van Doorn Kendrick van Doorn
Author Profile Icon Kendrick van Doorn
Kendrick van Doorn
Arrow right icon
View More author details
Toc

Table of Contents (20) Chapters Close

Preface 1. Part 1: Apache Airflow: History, What, and Why
2. Chapter 1: Getting Started with Airflow 2.0 FREE CHAPTER 3. Chapter 2: Core Airflow Concepts 4. Part 2: Airflow Basics
5. Chapter 3: Components of Airflow 6. Chapter 4: Basics of Airflow and DAG Authoring 7. Part 3: Common Use Cases
8. Chapter 5: Connecting to External Sources 9. Chapter 6: Extending Functionality with UI Plugins 10. Chapter 7: Writing and Distributing Custom Providers 11. Chapter 8: Orchestrating a Machine Learning Workflow 12. Chapter 9: Using Airflow as a Driving Service 13. Part 4: Scale with Your Deployed Instance
14. Chapter 10: Airflow Ops: Development and Deployment 15. Chapter 11: Airflow Ops Best Practices: Observation and Monitoring 16. Chapter 12: Multi-Tenancy in Airflow 17. Chapter 13: Migrating Airflow 18. Index 19. Other Books You May Enjoy

DAG deployments

We’ll start with a discussion of how to get your Airflow DAGs to your Airflow deployment. We won’t discuss specific implementation details because they can vary depending on your specific use case, security context, and operational needs.

Bundling

The DAG bundling pattern may be the most obvious pattern and is a very common starting point on your journey to using Airflow in production. It’s also antithetical to the code as configuration paradigm that is core to Airflow’s development and consumption.

The general pattern is to physically bundle your DAGs within the same file system as your Airflow system, even in a system with distributed compute. This is achieved by having your container build process install Airflow and copy your DAGs into the container during the image build process. This image can then be tagged and distributed to ensure that all components are operating using the exact same Airflow, plugin, and DAG versions.

...
lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at $19.99/month. Cancel anytime
Banner background image