Ingestion patterns
Ingesting data into the platform is still one of the key aspects of building any data platform – without data, there is no data platform.
It is a topic with a big range of possibilities as ingesting data depends on the data platform, the storage service, the file format, and many more big and small details.
In this chapter, we will concentrate on Snowflake, where it is possible to load data in multiple ways. However, the most efficient way to ingest decent amounts of data is through files. The concepts that we described for Snowflake apply, with some adaptations, to other platforms.
As an example, it is possible to use SQL insert statements, but the throughput is orders of magnitude lower than file-based ingestion or reading data from external tables (that, under the hood, read from files), but that also has its drawbacks, as we will see in a second.
In this section, we will present some of the most common use cases for ingesting data into Snowflake...
