Understanding IaC practices

IaC is a practice that consists of writing the code of the resources that make up an infrastructure.

This practice began to take effect with the rise of the DevOps culture and with the modernization of cloud infrastructure. Indeed, Ops teams that deploy infrastructures manually take the time to deliver infrastructure changes due to inconsistent handling and the risk of errors. Also, with the modernization of the cloud and its scalability, the way infrastructure is built requires reviewing the provisioning and change practices by adapting a more automated method.

IaC is the process of writing the code of the provisioning and configuration steps of infrastructure components, which helps automate its deployment in a repeatable and consistent manner.

Before we look at the use of IaC, we will see what the benefits of this practice are.

The benefits of IaC

The benefits of IaC are as follows:

• The standardization of infrastructure configuration reduces the risk of errors.
• The code that describes the infrastructure is versioned and controlled in a source code manager.
• The code is integrated into CI/CD pipelines.
• Deployments that make infrastructure changes are faster and more efficient.
• There's better management, control, and a reduction in infrastructure costs.

IaC also brings benefits to a DevOps team by allowing Ops to be more efficient in terms of infrastructure improvement tasks, rather than spending time on manual configuration. It also gives Dev the possibility to upgrade their infrastructures and make changes without having to ask for more Ops resources.

IaC also allows the creation of self-service, ephemeral environments that will give developers and testers more flexibility to test new features in isolation and independently of other environments.

IaC languages and tools

The languages and tools that are used to write the configuration of the infrastructure can be of different types; that is, scripting, declarative, and programmatic. We will explore them in the following sections.

Scripting types

These are scripts such as Bash, PowerShell, or others that use the different clients (SDKs) provided by the cloud provider; for example, you can script the provisioning of an Azure infrastructure with the Azure CLI or Azure PowerShell.

For example, here is the command that creates a resource group in Azure:

• Using the Azure CLI (the documentation is available at https://bit.ly/2V1OfxJ), we have the following:

  az group create --location westeurope --resource-group MyAppResourcegroup

• Using Azure PowerShell (the documentation is available at https://bit.ly/2VcASeh), we have the following:

  New-AzResourceGroup -Name MyAppResourcegroup -Location westeurope 

The problem with these languages and tools is that they require a lot of lines of code. This is because we need to manage the different states of the manipulated resources, and it is necessary to write all the steps of creating or updating the desired infrastructure.

However, these languages and tools can be very useful for tasks that automate repetitive actions to be performed on a list of resources (selection and query), or that require complex processing with certain logic to be performed on infrastructure resources, such as a script that automates VMs that carry a certain tag being deleted.

Declarative types

These are languages in which it is sufficient to write the state of the desired system or infrastructure in the form of configuration and properties. This is the case, for example, for Terraform and Vagrant from HashiCorp, Ansible, the Azure ARM template, Azure Bicep (https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/bicep-overview), PowerShell DSC, Puppet, and Chef. All the user has to do is write the final state of the desired infrastructure; the tool will take care of applying it.

For example, the following Terraform code allows you to define the desired configuration of an Azure resource group:

resource "azurerm_resource_group" "myrg" {
     name = "MyAppResourceGroup"
     location = "West Europe"
     tags = {
         environment = "Bookdemo"
    }
}

In this example, if you want to add or modify a tag, just modify the tags property in the preceding code and Terraform will do the update itself.

Here is another example that allows you to install and restart nginx on a server using Ansible:

---
- hosts: all
 tasks:
 - name: install and check nginx latest version
 apt: name=nginx state=latest
 - name: start nginx
 service:
 name: nginx
 state: started

To ensure that the service is not installed, just change the preceding code, with service as an absent value and the state property with the stopped value:

---
- hosts: all
 tasks:
 - name: stop nginx
 service:
 name: nginx
 state: stopped
 - name: check nginx is not installed
 apt: name=nginx state=absent

In this example, it was enough to change the state property to indicate the desired state of the service.

Note

For details regarding the use of Terraform and Ansible, see Chapter 2, Provisioning Cloud Infrastructure with Terraform, and Chapter 3, Using Ansible for Configuring IaaS Infrastructure.

Programmatic types

For a few years now, an observation has been made that the two types of IaC code, which are of the scripting or declarative languages, are destined to be in the operational team. This does not commonly involve the developers in the IaC.

This is done to create more union between developers and operations so that we see the emergence of IaC tools that are based more on languages known by developers, such as TypeScript, Java, Python, and C#.

Among the IaC tools that allow us to provision infrastructure using a programming language, we have Pulumi (https://www.pulumi.com/) and Terraform CDK (https://github.com/hashicorp/terraform-cdk).

The following is an example of some TypeScript code written with the Terraform CDK:

import { Construct } from 'constructs';
import { App, TerraformStack, TerraformOutput } from 'cdktf';
import {
  ResourceGroup,
} from './.gen/providers/azurerm';
class AzureRgCDK extends TerraformStack {
  constructor(scope: Construct, name: string) {
    super(scope, name);
    new AzurermProvider(this, 'azureFeature', {
      features: [{}],
    });
    const rg = new ResourceGroup(this, 'cdktf-rg', {
      name: 'MyAppResourceGroup',
      location: 'West Europe',
    });
  }
}
const app = new App();
new AzureRgCDK(app, 'azure-rg-demo');
app.synth();

In this example, which is written in Typescript, we are using two-tier libraries: the npm package and a Terraform CDK called cdktf. The npm package that's used to provision Azure resources is called 'gen/providers/azurerm'.

Then, we declare a new class that initializes the Azure provider and we define the creation of the resource group with the new ResourceGroup method.

Finally, to create the resource group, we instantiate this class and call the app.synth method of the CDK.

Note

For more information about the Terraform CDK, I suggest reading the following blog posts and watching the following video:

https://www.hashicorp.com/blog/cdk-for-terraform-enabling-python-and-typescript-support

https://www.hashicorp.com/blog/announcing-cdk-for-terraform-0-1

https://www.youtube.com/watch?v=5hSdb0nadRQ

The IaC topology

In a cloud infrastructure, IaC is divided into several typologies:

• Deploying and provisioning the infrastructure
• Server configuration and templating
• Containerization
• Configuration and deployment in Kubernetes

Let's deep dive into each topology.

Deploying and provisioning the infrastructure

Provisioning is the act of instantiating the resources that make up the infrastructure. They can be of the Platform-as-a-Service (PaaS) and serverless resource types, such as a web app, Azure function, or Event Hub, but also the entire network part that is managed, such as VNet, subnets, routing tables, or Azure Firewall. For virtual machine resources, the provisioning step only creates or updates the VM cloud resource, but not its content.

There are different provisioning tools we can use for this, such as Terraform, the ARM template, AWS Cloud training, the Azure CLI, Azure PowerShell, and also Google Cloud Deployment Manager. Of course, there are many more, but it is difficult to mention them all. In this book, we will look at, in detail, the use of Terraform to provide an infrastructure.

Server configuration

This step concerns configuring virtual machines, such as the hardening, directories, disk mounting, network configuration (firewall, proxy, and so on), and middleware installation.

There are different configuration tools, such as Ansible, PowerShell DSC, Chef, Puppet, and SaltStack. Of course, there are many more, but in this book, we will look in detail at the use of Ansible to configure a virtual machine.

To optimize server provisioning and configuration times, it is also possible to create and use server models, also called images, that contain all of the configuration (hardening, middleware, and so on) of the servers. While provisioning the server, we will indicate the template to use. So, in a few minutes, we will have a server that's been configured and is ready to be used.

There are also many IaC tools for creating server templates, such as Aminator (used by Netflix) and HashiCorp Packer.

Here is an example of some Packer file code for creating an Ubuntu image with package updates:

{
"builders": [{
     "type": "azure-arm",
     "os_type": "Linux",
     "image_publisher": "Canonical",
     "image_offer": "UbuntuServer",
     "image_sku": "16.04-LTS",
     "managed_image_resource_group_name": "demoBook",
     "managed_image_name": "SampleUbuntuImage",
     "location": "West Europe",
     "vm_size": "Standard_DS2_v2"
 }],
 "provisioners": [{
     "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
     "inline": [
     "apt-get update",
     "apt-get upgrade -y",
     "/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync"
 ],
 "inline_shebang": "/bin/sh -x",
 "type": "shell"
 }]
}

This script creates a template image for the Standard_DS2_V2 virtual machine based on the Ubuntu OS (the builders section). Additionally, Packer will update all the packages during the creation of the image with the apt-get update command. Afterward, Packer will deprovision the image to delete all user information (the provisioners section).

Note

The Packer part will be discussed in detail in Chapter 4, Optimizing Infrastructure Deployment with Packer.

Immutable infrastructure with containers

Containerization consists of deploying applications in containers instead of deploying them in VMs.

Today, it is very clear that the container technology to be used is Docker and that a Docker image is configured with code in a Dockerfile. This file contains the declaration of the base image, which represents the operating system to be used, additional middleware to be installed on the image, only the files and binaries necessary for the application, and the network configuration of the ports. Unlike VMs, containers are said to be immutable; the configuration of a container cannot be modified during its execution.

Here is a simple example of a Dockerfile:

FROM ubuntu
RUN apt-get update
RUN apt-get install -y nginx
ENTRYPOINT ["/usr/sbin/nginx","-g","daemon off;"]
EXPOSE 80

In this Docker image, we are using a basic Ubuntu image, installing nginx, and exposing port 80.

Note

The Docker part will be discussed in detail in Chapter 9, Containerizing Your Application with Docker.

Configuration and deployment in Kubernetes

Kubernetes is a container orchestrator – it is the technology that most embodies IaC (in my opinion) because of the way it deploys containers, the network architecture (load balancer, ports, and so on), and volume management, as well as how it protects sensitive information, all of which are described in the YAML specification files.

Here is a simple example of a YAML specification file:

apiVersion: apps/v1
kind: Deployment
metadata:
   name: nginx-demo
   labels:
       app: nginx
spec:
   replicas: 2
   selector:
     matchLabels:
        app: nginx
   template:
   metadata:
      labels:
        app: nginx
  spec:
    containers:
    - name: nginx
        image: nginx:1.7.9
      ports:
      - containerPort: 80

In the preceding specification file, we can see the name of the image to deploy (ngnix), the port to open (80), and the number of replicas (2).

Note

The Kubernetes part will be discussed in detail in Chapter 10, Managing Containers Effectively with Kubernetes.

IaC, like software development, requires that we implement practices and processes that allow the infrastructure code to evolve and be maintained.

Among these practices are those of software development, as follows:

• Have good principles of nomenclature.
• Do not overload the code with unnecessary comments.
• Use small functions.
• Implement error handling.

  Note

  To learn more about good software development practices, read the excellent book, which is, for my part, a reference on the subject, Clean Code, by Robert Martin.

However, there are more specific practices that I think deserve more attention:

• Everything must be automated in the code: When performing IaC, it is necessary to code and automate all of the provisioning steps and not leave the manual steps out of the code that distort the automation of the infrastructure, which can generate errors. And if necessary, do not hesitate to use several tools such as Terraform and Bash with the Azure CLI scripts.
• The code must be in a source control manager: The infrastructure code must also be in an SCM to be versioned, tracked, merged, and restored, and hence have better visibility of the code between Dev and Ops.
• The infrastructure code must be with the application code: In some cases, this may be difficult, but if possible, it is much better to place the infrastructure code in the same repository as the application code. This is to ensure we have better work organization between developers and operations, who will share the same workspace.
• Separation of roles and directories: It is good to separate the code from the infrastructure according to the role of the code. This allows you to create one directory for provisioning and configuring VMs and another that will contain the code for testing the integration of the complete infrastructure.
• Integration into a CI/CD process: One of the goals of IaC is to be able to automate the deployment of the infrastructure. So, from the beginning of its implementation, it is necessary to set up a CI/CD process that will integrate the code, test it, and deploy it in different environments. Some tools, such as Terratest, allow you to write tests on infrastructure code. One of the best practices is to integrate the CI/CD process of the infrastructure into the same pipeline as the application.
• The code must be idempotent: The execution of the infrastructure deployment code must be idempotent; that is, it should be automatically executable at will. This means that scripts must take into account the state of the infrastructure when running it and not generate an error if the resource to be created already exists, or if a resource to be deleted has already been deleted. We will see that declarative languages, such as Terraform, take on this aspect of idempotence natively. The code of the infrastructure, once fully automated, must allow the application's infrastructure to be constructed and destructed.
• To be used as documentation: The code of the infrastructure must be clear and must be able to serve as documentation. Infrastructure documentation takes a long time to write and, in many cases, it is not updated as the infrastructure evolves.
• The code must be modular: In infrastructure, the components often have the same code – the only difference is the value of their properties. Also, these components are used several times in the company's applications. Therefore, it is important to optimize the writing times of code by factoring it with modules (or roles, for Ansible) that will be called as functions. Another advantage of using modules is the ability to standardize resource nomenclature and compliance on some properties.
• Having a development environment: The problem with IaC is that it is difficult to test its infrastructure code under development in environments that are used for integration, as well as to test the application, because changing the infrastructure can have an impact. Therefore, it is important to have a development environment even for IaC that can be impacted or even destroyed at any time.

For local infrastructure tests, some tools simulate a local environment, such as Vagrant (from HashiCorp), so you should use them to test code scripts as much as possible.

Of course, the full list of good practices is longer than this; all the methods and processes of software engineering practices are also applicable.

Therefore, IaC, like CI/CD processes, is a key practice of DevOps culture that allows you to deploy and configure an infrastructure by writing code. However, IaC can only be effective with the use of appropriate tools and the implementation of good practices.

In this section, we covered an overview of some DevOps best practices. Next, we will present a brief overview of the evolution of the DevOps culture.

The evolution of the DevOps culture

With time and the experience that's been gained by using the DevOps culture, we can observe an evolution of the practices, as well as the teams that integrate with this movement.

This is, for example, the case of the GitOps practice, which is starting to emerge more and more in companies.

The GitOps workflow, which is commonly applied to Kubernetes, consists of using Git as the only source of truth; that is, the Git repository contains the code of the infrastructure state, as well as the code of the application to be deployed.

A controller will oversee retrieval of the Git source during a code commit, executing the tests, and redeploying the application.

Note

For more details about GitOps culture, practices, and workflows, read the official guide on the initiator of GitOps here: https://www.weave.works/technologies/gitops/.