Creating a serving layer

The final step in our pipeline will be a process to write to our serving layer, referred to as the Gold layer.

Refer to the following code:

private def writeDelta(df: DataFrame, tableName: String) = {
     df.write
       .format("delta")
       .mode("overwrite")
       .save(s"${target}${tableName}")
   }

This code defines a reusable function, writeDelta, that takes a DataFrame, df, and a table name, tableName, as arguments. It writes the DataFrame to a Delta Lake table with the specified table name, overwriting the existing data if it already exists.

Next, have a look at the following code:

val silverSource: String = "./src/main/scala/com/packt/dewithscala/chapter13/data/silver/ silver_devices"
val target: String = "./src/main/scala/com/packt/dewithscala...