You're reading from Distributed Data Systems with Azure Databricks Create, deploy, and manage enterprise data pipelines

Product type Paperback

Published in May 2021

Publisher Packt

ISBN-13 9781838647216

Length 414 pages

Edition 1st Edition

Languages

Python

Tools

Azure

Concepts

Data Science

Author (1):

Alan Bernardo Palacio

View More author details

Table of Contents (17) Chapters

Preface

1. Section 1: Introducing Databricks

2. Chapter 1: Introduction to Azure Databricks FREE CHAPTER

3. Chapter 2: Creating an Azure Databricks Workspace

4. Section 2: Data Pipelines with Databricks

5. Chapter 3: Creating ETL Operations with Azure Databricks

6. Chapter 4: Delta Lake with Azure Databricks

7. Chapter 5: Introducing Delta Engine

8. Chapter 6: Introducing Structured Streaming

9. Section 3: Machine and Deep Learning with Databricks

10. Chapter 7: Using Python Libraries in Azure Databricks

11. Chapter 8: Databricks Runtime for Machine Learning

12. Chapter 9: Databricks Runtime for Deep Learning

13. Chapter 10: Model Tracking and Tuning in Azure Databricks

14. Chapter 11: Managing and Serving Models with MLflow and MLeap

15. Chapter 12: Distributed Deep Learning in Azure Databricks

16. Other Books You May Enjoy

pandas DataFrame API (Koalas)

Data scientists and data engineers that are Python users are very familiar with working with pandas DataFrames when manipulating data. pandas is a Python library for data manipulation and analysis but that lacks the capability to work with big data, therefore it is only suitable when working with small datasets. When we need to work with more data, the most common option is PySpark, as we have demonstrated in the previous section, which is a library with a very different syntax than pandas.

Koalas is a library that eases the learning curve from transitioning from pandas to working with big data in Azure Databricks. Koalas has a syntax that is very similar to the pandas API but with the functionality of PySpark.

Not all the pandas methods have been implemented and there are many small differences or subtleties that must be considered and might not be obvious. We cannot understand Koalas without understanding PySpark.

Koalas, functionality is built...