You're reading from Data Engineering with AWS Cookbook A recipe-based approach to help you tackle data engineering problems with AWS services

Product type Paperback

Published in Nov 2024

Publisher Packt

ISBN-13 9781805127284

Length 528 pages

Edition 1st Edition

Languages

Python

Tools

AWS Glue

Concepts

Data Engineering Cloud Computing

Authors (4):

Viquar Khan

Gonzalo Herreros González

Huda Nofal

Trâm Ngọc Phạm

View More author details

Table of Contents (16) Chapters

Preface

1. Chapter 1: Managing Data Lake Storage

2. Chapter 2: Sharing Your Data Across Environments and Accounts FREE CHAPTER

3. Chapter 3: Ingesting and Transforming Your Data with AWS Glue

4. Chapter 4: A Deep Dive into AWS Orchestration Frameworks

5. Chapter 5: Running Big Data Workloads with Amazon EMR

6. Chapter 6: Governing Your Platform

7. Chapter 7: Data Quality Management

8. Chapter 8: DevOps – Defining IaC and Building CI/CD Pipelines

9. Chapter 9: Monitoring Data Lake Cloud Infrastructure

10. Chapter 10: Building a Serving Layer with AWS Analytics Services

11. Chapter 11: Migrating to AWS – Steps, Strategies, and Best Practices for Modernizing Your Analytics and Big Data Workloads

12. Chapter 12: Harnessing the Power of AWS for Seamless Data Warehouse Migration

13. Chapter 13: Strategizing Hadoop Migrations – Cost, Data, and Workflow Modernization with AWS

14. Index

Why subscribe?

15. Other Books You May Enjoy

Migrating the Hive Metastore to AWS

The Hive Metastore is a crucial component within Hadoop, acting as a centralized storehouse for metadata related to Hive tables, schemas, and partitions. When transitioning your Hadoop cluster to the AWS cloud, you can opt to either establish a dedicated Hive Metastore on AWS or utilize the managed AWS Glue Data Catalog.

Migrating to the AWS Glue Data Catalog provides benefits such as schema versioning and efficient integration with Amazon EMR, especially for transient clusters. This migration ensures high availability, fault tolerance, and better data governance.

When it comes to data discovery and management, two data catalogs are among the most popular choices:

Hive Metastore: This repository houses essential information about Hive tables and their underlying data structures, including partition names and data types. Hive, an open source data warehousing and analytics tool built on Hadoop, can be deployed on platforms such as EMR...

The rest of the chapter is locked

You're reading from Data Engineering with AWS Cookbook A recipe-based approach to help you tackle data engineering problems with AWS services

Table of Contents (16) Chapters

Migrating the Hive Metastore to AWS

Unlock this book and the full library FREE for 7 days

Authors (4)

Personalised recommendations for you