You're reading from Data Engineering with AWS Cookbook A recipe-based approach to help you tackle data engineering problems with AWS services

Product type Paperback

Published in Nov 2024

Publisher Packt

ISBN-13 9781805127284

Length 528 pages

Edition 1st Edition

Languages

Python

Tools

AWS Glue

Concepts

Data Engineering Cloud Computing

Authors (4):

Viquar Khan

Gonzalo Herreros González

Huda Nofal

Trâm Ngọc Phạm

View More author details

Table of Contents (16) Chapters

Preface

1. Chapter 1: Managing Data Lake Storage

2. Chapter 2: Sharing Your Data Across Environments and Accounts FREE CHAPTER

3. Chapter 3: Ingesting and Transforming Your Data with AWS Glue

4. Chapter 4: A Deep Dive into AWS Orchestration Frameworks

5. Chapter 5: Running Big Data Workloads with Amazon EMR

6. Chapter 6: Governing Your Platform

7. Chapter 7: Data Quality Management

8. Chapter 8: DevOps – Defining IaC and Building CI/CD Pipelines

9. Chapter 9: Monitoring Data Lake Cloud Infrastructure

10. Chapter 10: Building a Serving Layer with AWS Analytics Services

11. Chapter 11: Migrating to AWS – Steps, Strategies, and Best Practices for Modernizing Your Analytics and Big Data Workloads

12. Chapter 12: Harnessing the Power of AWS for Seamless Data Warehouse Migration

13. Chapter 13: Strategizing Hadoop Migrations – Cost, Data, and Workflow Modernization with AWS

14. Index

Why subscribe?

15. Other Books You May Enjoy

Migrating an Oozie database to the Amazon RDS MySQL

Apache Oozie, a widely used workflow scheduler in the Hadoop ecosystem, orchestrates a variety of Hadoop jobs, including Hive, Pig, Sqoop, Spark, DistCp, Linux shell actions, and more. It stands out in the Hadoop community for its scalability and reliability.

Oozie operates with two key components: workflow jobs, which allow you to map out workflow steps in the form of Directed Acyclic Graphs (DAGs), and the Oozie Coordinator, designed for scheduling these workflow jobs based on events or timed triggers.

Using XML definitions, Oozie enables the creation of workflows and has been available on Amazon EMR since the 5.0.0 release.

Like Hive, Oozie also relies on a Metastore database, a crucial aspect to consider during migration. When moving Oozie workflows to EMR, it’s essential to transfer both the workflow definition files and the Metastore database.

This recipe provides a step-by-step walkthrough of migrating your...

The rest of the chapter is locked

You're reading from Data Engineering with AWS Cookbook A recipe-based approach to help you tackle data engineering problems with AWS services

Table of Contents (16) Chapters

Migrating an Oozie database to the Amazon RDS MySQL

Unlock this book and the full library FREE for 7 days

Authors (4)

Personalised recommendations for you