This chapter and the remainder of this book require you to have a Google Cloud account. You can use an existing Google Cloud project, but we recommend creating a new clean project for you to follow along. You should also have the Google Cloud command-line interface (CLI) installed on your local PC and be familiar with basic gcloud commands. Please see https://cloud.google.com/sdk/docs/install for detailed instructions on how to download the CLI.

Of course, we are using Terraform. Terraform is available on all common operating systems and is easy to install. You can download the version for your operating system at https://www.terraform.io/downloads. HashiCorp is constantly improving the tool by providing updates and upgrades. For the writing of this book, we are using v1.3.3. The code should work with any version greater than v.1.3.0. However, we suggest you download this particular version if you run into any issues.

The source code for this chapter and all other chapters is available at https://github.com/PacktPublishing/Terraform-for-Google-Cloud-Essential-Guide.

We recommend that you download the code, enabling you to follow along. We organized the code into analogous chapters and sections and indicated the appropriate subdirectories.

The rise of cloud computing since mid-2000 has been spectacular. Hardly a month goes by without the three hyperscalers (Amazon Web Services (AWS), Azure, and Google Cloud) announcing the opening of a new data center region. Cloud computing—in particular, public cloud—offers an incredible array of technologies at a nearly infinite scale. This has led to a new way of deploying and operating IT. DevOps combines two areas that were traditionally distinct phases in software development—development and operations. DevOps aims to deliver software at a much faster pace than the traditional Waterfall model was able to. By combining historically distinct phases and teams, DevOps methodology can deliver software much more rapidly and with higher quality than traditional software methodology.

One key aspect of DevOps is automation. Combining several separate tools into a pipeline, we can deliver software from development to production with minimal human intervention. This concept of continuous integration and continuous delivery, usually referred to as CI/CD, integrates managing source code, provisioning the IT infrastructure, compiling (if necessary), packaging, testing, and deploying into a pipeline. A CI/CD pipeline requires automation at every step to execute efficiently.

Infrastructure as Code

Automating the provisioning of the IT infrastructure is a key component of a CI/CD pipeline and is known as IaC. In traditional on-prem computing, servers and networking infrastructure provision was a long-drawn and manual process. It started with ordering IT hardware, the physical setup of the hardware, and configuration, such as installing the operating system and configuring the network infrastructure. This process would often take weeks or months. Virtualization somewhat helped the process, but the provisioning of the infrastructure would generally still fall onto a separate team before developers could start deploying the fruit of their labor into a test environment, much less a production one.

The rise of cloud computing shortened this process from months to days and even hours. Infrastructure can now be provisioned through a graphical user interface (GUI) known as the web console. Initially, this was a straightforward process as the number of services and configuration options were manageable. However, as more services became available and the configuration options increased exponentially, provisioning networks, servers, databases, and other managed services became tedious and error-prone.

A key objective in DevOps is to have essentially identical environments. That is, the development, test, and production environments should be the same except for some minor configuration changes; for example, the database in development and production should be identical except for the number of CPUs and the size of memory.

However, configuring complex environments using the web console is tedious and error prone. Using what is sometimes derided as ClickOps, the provisioning of even a medium-complex environment requires hundreds, if not thousands, of user interactions or clicks.

Using a CLI can help with configuring multiple environments. One can develop scripts that combine multiple CLI commands into a single executable. However, managing and making changes is next to impossible using a CLI.

Enter IaC. IaC is the provisioning of infrastructure using code rather than a web console or a CLI. For example, you can write code to achieve the same step instead of using the web console and going through several manual steps to provision a server. For example, the main.tf file featured in this chapter shows the configuration of a server with the Debian operating system. The server, called cloudshell, is an e2-micro instance placed in the us-central1-a region.

Once you have the code, you can reuse it repeatedly. That is, you can deploy many servers with minimal or no change. Since, it is regular code, you can use version control and source code revision systems to manage your code. This facilitates effective teamwork.

Furthermore, IaC can be integrated into a CI/CD pipeline, including testing and validation. That is, you can validate and test the provisioning of servers before they are deployed. Thus, you can provision complex infrastructure involving hundreds of servers, complex networking, and multiple services within minutes rather than days or weeks. Ultimately, this makes the software development release process faster and more secure, which is precisely the objective of DevOps.

A CI/CD pipeline includes many steps besides IaC. One of the steps is configuration management, which includes configuring servers such as installing updates, libraries, and code. Ansible, Puppet, and Chef are common configuration management tools. Some overlap exists between configuration management tools and IaC tools. Configuration management tools can provision some infrastructure, while IaC tools perform some configuration tasks. However, infrastructure provisioning and configuration management should generally be considered two separate steps better served by different tools.

Terraform

Terraform has become the most popular IaC tool in recent years. While each cloud platform has its proprietary IaC tool (AWS has CloudFormation, Azure has Azure Resource Manager, and Google Cloud has Deployment Manager), Terraform is unique in that it is platform agnostic. You can use Terraform to provision infrastructure in any cloud provider and for many other platforms such as vSphere and Google Workspace.

As you will see, Terraform is easy to learn and provides a straightforward workflow that can be easily incorporated into any CI tool. Terraform is open source and has developed an active community with its own ecosystem. Community members have developed tools that help you write Terraform code more effectively; we cover some of them in this book.

Note

The code for this section is under chap01/cloudshell in the GitHub repo of this book.

The easiest way to run Terraform in Google Cloud is using Google Cloud Shell, which is a preconfigured development environment accessible through your browser. It comes pre-installed with the latest version of common utilities, including the latest version of Terraform. Furthermore, all authentication is already set up. So, let’s give it a try.

Note

You can check the current version of Terraform using the terraform –version command.

If your project is new, and you have not provisioned a virtual machine (VM) (Compute Engine), run the following gcloud command to enable the compute API:

$ gcloud services enable compute.googleapis.com

Then place the following code in a file called main.tf. This is known as the Terraform configuration. Configurations are written in HashiCorp Configuration Language (HCL). HCL is human readable and is used by several HashiCorp tools.

Note

In this book, we will use the term Terraform language to refer to the language in which the configuration files are written.

This book aims to make you proficient in Terraform to write high-quality, reusable code to provision resources in Google Cloud efficiently and securely.

The file’s actual name is arbitrary, but note that the .tf extension is mandatory. Terraform uses the .tf extension to identify Terraform configuration files.

main.tf

resource "google_compute_instance" "this" {
  name         = "cloudshell"
  machine_type = "e2-small"
  zone         = "us-central1-a"
  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network = "default"
  }
}

Next, run the following two commands:

$ terraform init
$ terraform apply

Terraform will ask you where you want to perform these actions, so type in yes to approve.

Congratulations—you have now provisioned your first server using Terraform! Go to the web console and inspect the compute engine that Terraform has created.

As expected, Terraform created a Debian server named cloudshell of a machine type e2-small in the default network:

Figure 1.1 – Terraform-provisioned server in the web console

Let’s have a more detailed look at how to write Terraform configurations and how Terraform operates.

As mentioned, the Terraform language is based on HCL. The most basic construct is a block, defined as a “container for other content” (https://developer.hashicorp.com/terraform/language/syntax/configuration#blocks). For example, the preceding code block is of the type resource. A resource block has two labels. The first label consists of the provider and resource name. For example, the resource label for a Google compute engine is google_compute_instance, whereas the label for an AWS E2 instance is aws_instance. The second label in a resource block is the ID, the name you give so that Terraform can uniquely identify each resource. Please note that no two resources can have the same name. So, if you want to provision two VMs, you define two resource blocks, each of type google_compute_instance, and each with a unique name.

The body of the block is nested within {}. Blocks can have nested blocks. In the preceding code example, there are two nested blocks, boot_disk (which in itself has a nested block called initialize_params) and network_interface.

While the syntax for Terraform is the same, each cloud provider has its unique resources and data source definitions. The Terraform documentation for Google Cloud is at https://registry.terraform.io/providers/hashicorp/google/latest/docs. You see that all resources and data sources start with the google keyword. In general, it is followed by the name of the equivalent gcloud command. Thus, the resource to create a compute instance is named google_compute_instance, whereas the resource to provision a SQL database instance is named google_sql_database_instance.

Looking at the documentation, you can see the current Google services that are currently supported. More are added as Google Cloud introduces new services. We recommend you bookmark the documentation as you have to refer to it frequently.

In the first part of this book, we use only a few Google Cloud resources as we want to concentrate on the language. We design complex architectures using additional Google Cloud services in the second part. But first, let’s look at the steps involved in using Terraform.

A basic Terraform workflow consists of the following two steps:

1. Init
2. Apply

In this first step, terraform init initializes Terraform and downloads all the necessary configuration files. You see that after init is run, there is a hidden directory named .terraform. This directory is where Terraform stores various configuration files. You must run the initialization step every time you write a new configuration file. Don’t worry; you can run this command multiple times, and you learn over time when you need to rerun it. If in doubt, run it again as it only takes a few seconds.

The second step, terraform apply, consists of two phases. First, Terraform creates an execution plan, then it executes this plan. The execution plan is an internal plan of the actions Terraform will perform and in which order. Terraform outputs this plan, including a summary of how many resources will be added, changed, and destroyed. You should always review this output carefully. Once you have confirmed that the plan does what you intended, you can confirm, and Terraform actually provisions the cloud resources.

You can also run the two phases explicitly by running terraform plan, saving the plan in a file, and then running terraform apply against that file, like so:

$ terraform plan --out=plan
$ terraform apply plan

This is often done in CI/CD pipelines, and we will return to it later.

One of the advantages of IaC is that you can quickly remove resources. This is not only useful during the development phase, as you can test things quickly, but it also can help you save costs. At the end of the workday, you can remove all the resources and reprovision them the next day. To destroy resources in Terraform, simply execute the following command:

$ terraform destroy

Now, after you have run terraform destroy and run terraform apply twice, you’ll notice that the second time you run terraform apply, Terraform reports no changes, and no action is performed. The next chapter discusses how Terraform decides what actions to take.

There are many more Terraform commands, and we introduce them throughout the book. For now, remember the following four commands:

• terraform init to initialize
• terraform plan to view and create a Terraform plan
• terraform apply to actually provision the resources
• terraform destroy to remove all resources

Now that we have described the basic workflow of Terraform and shown how you run Terraform in Google Cloud Shell, let’s look at the different ways of running Terraform on your local computer.

While running Terraform in the Cloud Shell is the easiest way to run Terraform, most developers prefer to edit and develop code on their own machine in their local environment.

Terraform provides an easy way to do this. First, install Terraform on your computer. Terraform is available on most platforms, including Linux, macOS, and Windows. To install Terraform, please go to https://developer.hashicorp.com/terraform/downloads and follow the installation instructions for your environment.

In this book, we use Terraform on Ubuntu, but the code and the principles are the same regardless of the operating system.

You also require the Google Cloud CLI: gcloud. So, if you have not already installed it, please go ahead and do so. To see whether you have successfully installed both, run these commands in your preferred shell on your local machine:

$ gcloud --version
$ terraform –version

Once you have successfully installed both gcloud and Terraform, you need to authenticate Terraform against the Google Cloud project you are using.

Authentication using a service account

Note

The code for this section is in chap01/key-file.

There are several ways to authenticate Terraform with Google Cloud. The first is using a service account. For this, you create a service account and download a key file.

You can create a service account and download the key file interactively using the web console or the following gcloud commands. If you haven’t authenticated against Google Cloud in your shell, please do so now using the gcloud auth login command.

Please note that for simplicity’s sake, we have given the Terraform service account the broad role of editor. This is not recommended as it violates the security principle of least privilege. You need to replace <PROJECT_ID> with the ID of your Google Cloud project:

$ gcloud auth login --no-launch-browser
$ gcloud config set project "<PROJECT_ID>"
$ gcloud iam service-accounts create terraform \
    --description="Terraform Service Account" \
    --display-name="terraform"
$ export GOOGLE_SERVICE_ACCOUNT=`gcloud iam service-accounts \
 list --format="value(email)"  --filter=name:"terraform@"` 
$ export GOOGLE_CLOUD_PROJECT=`gcloud info \
--format="value(config.project)"`
$ gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
    --member="serviceAccount:$GOOGLE_SERVICE_ACCOUNT" \
    --role="roles/editor"
$ gcloud iam service-accounts keys create "./terraform.json"  \
  --iam-account=$GOOGLE_SERVICE_ACCOUNT

Once you have a key file, you can reference it in your Terraform configuration using the provider declaration.

As we mentioned, Terraform does not care about filenames. It combines all files with the .tf extension and then creates an execution plan using all the configurations. However, while Terraform doesn’t care about filenames, humans do. As one of the objectives of IaC is to share and reuse code, there are commonly adopted file naming conventions. We use those conventions throughout the book.

The convention is to place the provider declaration in a file called provider.tf. We mentioned earlier that Terraform supports several cloud providers and other infrastructure tools. It does so by using the concept of a provider. A provider is a translation layer between Terraform and the external API of the provider. When you run Terraform, it translates your configuration code into API calls that interact with the provider—in this case, Google Cloud. If you run Terraform with the AWS provider, it generates AWS API calls.

This is also the reason that while Terraform works with all cloud providers, each configuration file is specific to a particular cloud provider. You cannot use the same code to provision a Google Cloud and an AWS VM. However, the principles of developing and managing Terraform code and the workflow are similar for all cloud providers. Thus, if you have mastered Terraform for Google Cloud, you can use that knowledge to learn Terraform for AWS and Azure.

The Google provider has several optional attributes. The most commonly used attributes are project, region, and zone, which specify the default project, region, and zone. So, the provider file for our example using the credentials file in the current directory looks like this:

Note

You need to replace <PROJECT_ID> with the project ID of your Google Cloud project.

provider.tf

provider "google" {
  project     = <PROJECT_ID>
  region      = "us-central1"
  zone        = "us-central1-c"
  credentials = "./terraform.json"
}

Thus, you can now run the Terraform workflow on your local computer, like so:

$ terraform init
$ terraform apply

Service account keys are very powerful in Google Cloud and must be treated carefully. Thus, we do not recommend including a reference to the credentials and—particularly—storing the key file in the local directory. It is easy to accidentally include the key file in your source code, where it can leak. It also makes the code less portable.

Authentication using a service account and environment variable

Note

The code for this section is in chap01/environment-variable.

A better way is to store the key content in an environment variable, which can, in turn, be stored more securely (for example, via envchain) than plaintext key files on disk, as can be seen in the following command:

$ export GOOGLE_CREDENTIALS=`cat ../key-file/terraform.json`

Using an environment variable has the added benefit of making your Terraform code more portable, as it does not include the authentication method in the Terraform code.

Service account impersonation

Note

The code for this section is in chap01/service-account-impersonation.

A third method is to use service account impersonation. This Google Cloud concept lets users act as or impersonate a service account. To use service account impersonation, you first need to allow you, the user, the Identity and Access Management (IAM) role of Service Account Token Creator (roles/iam.serviceAccountTokenCreator). Please note that this role is not enabled by default, even for a project owner, so you have to add that IAM role explicitly.

Second, set the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable to the Terraform service account we created in the previous section. Ensure that the GOOGLE_APPLICATION_CREDENTIALS environment variable is NOT set.

Next, acquire the user credentials to use for Application Default Credentials (ADC) using the following command:

$ gcloud auth application-default login

These are credentials that Terraform uses to impersonate the service account and perform other actions on behalf of that service account using the IAM permission of that service account. Please note that this is a separate authentication from the Google platform credentials (gcloud auth login).

As we mentioned, service account keys must be handled with care, thus it is better to delete them if they are not in use. Thus, here is the complete code to impersonate the service account and clean up the environment:

# Delete unused credentials locally
$ unset GOOGLE_CREDENTIALS
# Delete previously issued keys
$ rm ../key-file/terraform.json
$ gcloud iam service-accounts keys list \
--iam-account=$GOOGLE_ SERVICE_ACCOUNT
$ gcloud iam service-accounts keys delete \
<SERVICE ACCOUNT_ID> --iam-account=$GOOGLE_SERVICE_ACCOUNT
# Set up impersonation
$ export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=\
`gcloud iam service-accounts list --format="value(email)" \
--filter=name:terraform`
$ gcloud auth application-default login --no-launch-browser
$ export USER_ACCOUNT_ID=`gcloud config get core/account`
$ gcloud iam service-accounts add-iam-policy-binding \
    $GOOGLE_IMPERSONATE_SERVICE_ACCOUNT \
    --member="user:$USER_ACCOUNT_ID" \
    --role="roles/iam.serviceAccountTokenCreator"
$ export GOOGLE_CLOUD_PROJECT=`gcloud info \ --format="value(config.project)"`

While this may look tedious, you only need to do it once. Using impersonation makes the code more portable and more secure, as it does not rely on service account keys and does not specify the authentication method in your configuration files. Please note that it might take a couple of minutes for the added IAM role to become effective. But once it is effective, run the following command:

$ terraform apply

Now that we have shown different ways to authenticate Terraform to Google Cloud and created a simple server, let’s provision something more useful. But before we move to the next section, don’t forget to destroy the server by running the following command:

$ terraform destroy

Note

The code for this section is in chap01/parameterizing-terraform.

So far, we have provisioned a server that doesn’t really do anything. To conclude this first chapter, let’s expand on it to demonstrate the power of IaC. First, we add variables to make our code more generic. In Terraform, you need to declare variables in a variable block. While you can declare variables anywhere in your code, by convention, it is best to declare them in a file called variables.tf. No argument is required for a variable declaration, but it is a good idea to define the type and a description and, if useful, to add a default value. You can also add validation and specify that the variable contains a sensitive value, but more on that later:

variables.tf

variable "project_id" {
  type        = string
  description = "ID of the Google Project"
}
variable "region" {
  type        = string
  description = "Default Region"
  default     = "us-central1"
}
variable "zone" {
  type        = string
  description = "Default Zone"
  default     = "us-central1-a"
}
variable "server_name" {
  type        = string
  description = "Name of server"
}
variable "machine_type" {
  type        = string
  description = "Machine Type"
  default  = "e2-micro"
}

Variables are referenced using the var.<variable_name> syntax. Thus, our parameterized main.tf file now looks like this:

main.tf

resource "google_compute_instcance" "this" {
  name         = var.server_name
  machine_type = var.machine_type
  zone         = var.zone
  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }
  network_interface {
    network = "default"
    access_config {
      // Ephemeral public IP
    }
  }
  metadata_startup_script = file("startup.sh")
  tags = ["http-server"]
}

We made two minor additions to the google_compute_instance resource. We added the access_config block to the network_interface block, which assigns a public IP address to the server, and added an http-server network tag. This allows HTTP traffic to reach the server using the default-allow-http firewall rule. (Please note that this firewall rule is created the first time you provision a compute instance and enable Allow HTTP traffic in the web console or in gcloud. Thus, if you haven’t done so in your current project, please do so, as Google Cloud automatically creates this firewall rule. Later, we show how to create firewall rules using Terraform).

There are multiple ways to assign a value to a variable. First, you can specify a default value in the declaration. Second, you can pass it as a value either interactively or via the command-line flag. One of the common ways to specify variable values is in the variable definitions file or .tfvars file. This file contains only variable assignments for the form variable = value form. You will learn the usefulness of tfvars in the third chapter:

terraform.tfvars

project_id = <PROJECT_ID>
server_name = "parameterizing-terraform"

Note

You need to replace <PROJECT_ID> with the ID of your Google Cloud project.

The second change we can make is to configure a web server automatically. As we said earlier, there is some overlap between IaC and configuration management. You can include startup scripts to perform some configuration code once the server is deployed. Configuration management tools provide much more functionality, and you should use the startup scripts only for basic configuration. Alternatively, you can use the startup script to run the configuration management tool. Our startup script installs the Apache web server and adds the obligatory “Hello World” text:

startup.sh

#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Hello World!</p></body></html>

Lastly, we can use the output block to output the public IP address of the server (don’t worry about the syntax—we elaborate on that later). Again, we use the convention to place the output block in a file named outputs.tf:

outputs.tf

output "instance_ip_addr" {
  value = google_compute_instance.this.network_interface.0.access_config.0.nat_ip
}

Thus, you should now have the following files in your current directory:

├── main.tf ├── outputs.tf ├── provider.tf ├── startup.sh ├── terraform.tfvars └── variables.tf

Terraform provisions the server, and then output its IP address. Copy the IP address and paste it into a browser. If you get a timeout error, ensure that the default-allow-http firewall rule is set.

While we defined the default machine type as e2-micro, we can override any variable value on the command line using the -var flag.

Thus, the following command provisions the equivalent server but with an e2-small machine type:

$ terraform destroy
$ terraform apply -var machine_type=e2-small

As we conclude this chapter, it is a good idea to clean up your environment and remove servers and resources you don’t need anymore. So, if you have not done so, run the following command:

$ terraform destroy

Confirming that you removed all unnecessary servers using the web console is also a good practice.

This chapter presented four different methods of running and authenticating Terraform in Google Cloud. The first one is to run Terraform in Google Cloud Shell, which requires no installation or authentication. To run Terraform locally, you need to use a service account using either a key file or service account impersonation.

Managing key files poses a security risk. Key files are not automatically rotated and hence tend to be long-lived. Even if your organization manually rotates key files, they then need to be distributed, which introduces considerable overhead.

Using service account impersonation eliminates the risk associated with generating and distributing service account keys. Service account impersonation also makes code more portable as it does not depend on any external file.

In this chapter, we introduced the concept of IaC and why it is critical to DevOps. IaC makes the provisioning of cloud resources faster, more consistent, and, ultimately, more secure. There are several IaC tools, but Terraform has emerged as the de facto IaC tool for it is easy to use and, through the concept of providers, can be used with any cloud provider.

You provisioned your first resource in Google Cloud using Terraform in Google Cloud Shell. But as most cloud engineers and architects prefer to use their own machines, you learned three authentication methods for using Terraform on your local PC while provisioning resources in your Google Cloud project. We showed why service account impersonation is the preferred method of authenticating Terraform by eliminating the risks associated with service account keys.

The chapter also introduced variables to make Terraform configurations more flexible and showed the different methods of assigning values to variables.

In the next chapter, you deepen your understanding of Terraform by delving into the inner workings, discussing the concept of the Terraform state, and introducing additional concepts of the Terraform language.

Before you go to the next chapter, make sure that you have removed all servers not to incur any unnecessary costs. The beauty of IaC and Terraform is that you only need to rerun Terraform to recreate resources if you need them again.