You're reading from Data Ingestion with Python Cookbook A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process

Product type Paperback

Published in May 2023

Publisher Packt

ISBN-13 9781837632602

Length 414 pages

Edition 1st Edition

Languages

Python

Tools

Apache Airflow

Concepts

Data Engineering

Author (1):

Gláucia Esppenchutz

View More author details

Table of Contents (17) Chapters

Preface

1. Part 1: Fundamentals of Data Ingestion

2. Chapter 1: Introduction to Data Ingestion FREE CHAPTER

3. Chapter 2: Principals of Data Access – Accessing Your Data

4. Chapter 3: Data Discovery – Understanding Our Data before Ingesting It

5. Chapter 4: Reading CSV and JSON Files and Solving Problems

6. Chapter 5: Ingesting Data from Structured and Unstructured Databases

7. Chapter 6: Using PySpark with Deﬁned and Non-Deﬁned Schemas

8. Chapter 7: Ingesting Analytical Data

9. Part 2: Structuring the Ingestion Pipeline

10. Chapter 8: Designing Monitored Data Workﬂows

11. Chapter 9: Putting Everything Together with Airﬂow

12. Chapter 10: Logging and Monitoring Your Data Ingest in Airﬂow

13. Chapter 11: Automating Your Data Ingestion Pipelines

14. Chapter 12: Using Data Observability for Debugging, Error Handling, and Preventing Downtime

15. Index

Why subscribe?

16. Other Books You May Enjoy

Ingesting Parquet ﬁles

Apache Parquet is a columnar storage format that is open source and designed to support fast processing. It is available to any project in a Hadoop ecosystem and can be read in different programming languages.

Due to its compression and fastness, this is one of the most used formats when needing to analyze data in great volume. The objective of this recipe is to understand how to read a collection of Parquet files using PySpark in a real-world scenario.

Getting ready

For this recipe, we will need SparkSession to be initialized. You can use the code provided at the beginning of this chapter to do so.

The dataset for this recipe will be Yellow Taxi Trip Records from New York. You can download it by accessing the NYC Government website and selecting 2022 | January | Yellow Taxi Trip Records or using this link:

https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-01.parquet

Feel free to execute the code with a Jupyter notebook...