Deploying a data pipeline using Terraform

In this recipe, you will see the Terraform cloud-agnostic IaC automation tool. Agnostic doesn’t mean you can abstract yourself from the underlying vendor but that the tool has support for multiple cloud providers, so you can apply the knowledge of the tool to other providers as well as build multi-cloud projects. But you still need to know about the different services on each cloud provider and how to use them. See the official vendor documentation for further details: https://www.terraform.io/.

The trade-off of its multi-cloud support is that because it’s not provided directly by each vendor, it may take some time until Terraform supports new features.

Getting ready

To complete this recipe, you need a bash command line with the AWS CLI set up, as indicated in the Technical requirements section at the beginning of this chapter.

You also need the Terraform executable in your system. You must put it in the system’...