Introduction to CI/CD, along with a branching strategy

In this section of the chapter, we will dig into what exactly CI/CD is and why is it so important in the software life cycle. Then, we will learn about a branching strategy, and how we use it in the source code repository to make the software delivery more efficient, collaborative, and faster.

CI

Before getting to know about CI, let's have a brief look at what happens in a software development workflow. Suppose you are working independently, and you have been asked to develop a web application that is a chat system. So, the first thing you will be doing is to create a Git repository and write your code in your local machine, build the code, and run some tests. If it works fine in your local environment, you will then push it to a remote Git repository. After that, you will build this code for different environments (where the actual application will run) and put the artifact in the artifact registry. After that, you will deploy that artifact into the application server where your application will be running.

Now, suppose the frontend of your application is not too good, and you want some help from your frontend developer. The frontend developer will clone the code repository, then contribute to the repository either by modifying the existing code or adding new code. After that, they will commit the code and push it into the repository. Then again, the same steps of build and deploy will take place, and your application will be running with the new User Interface (UI). Now, what if you and the frontend developer both want to enhance the application, whereby both of you will be writing the code, and somehow you both used the same file and did your own changes, and tried to push back to the repository? If there are no conflicts, then your Git repository will allow you to update the repository, but in case there are any conflicts then it will highlight this to you. Now, once your code repository is updated, you must again build the code and run some unit tests. If the tests find a bug, then the build process will fail and you or the frontend developer will need to fix the bug, and again run the build and unit test. Once this passes, you will then need to put the build artifact into the artifact registry and then later deploy it into the application server. But this whole manual process of building, testing, and making the artifact ready for deployment will become quite troublesome and slow when your application gets bigger, and collaborators will increase, which in return will slow the deployment of your application. These problems of slow feedback and a manual process will easily put the project off schedule. To solve this, we have a CI process.

CI is a process where all the collaborators/developers contribute their code, several times a day, in a central repository that is further integrated into an automated system that pulls the code from the repository, builds it, runs unit tests, fails the build, gives feedback in case there are bugs, and prepares the artifact so that it is deployment-ready. The process is illustrated in the following diagram:

Figure 1.1 – CI process

CI makes sure that software components or services work together. The integration process should take place and complete frequently. This increases the frequency of developer code commits and reduces the chances of non-compatible code and redundant efforts. To implement a CI process, we need—at the very least—the following tools:

• Version Control System (VCS)
• Build tool
• Artifact repository manager

While implementing a CI process, our code base must be under version control. Every change applied to the code base must be stored in a VCS. Once the code is version controlled, it can be accessed by the CI tool. The most widely used VCS is Git, and in this book, we will also be using Git-based tools. The next requirement for CI is a build tool, which basically compiles your code and provides the executable file in an automated way. The build tool depends on the technology stack; for instance, for Java, the build tool will be Maven or Ant, while for Node.js, it will be npm. Once an executable file gets generated by the build tool, it will be stored in the artifact repository manager. There are lots of tools available in the market—for example, Sonatype Nexus Repository Manager (NXRM) or JFrog. We will be using the AWS Artifact service. The whole CI workflow will be covered in detail after the Branching strategy (Gitflow) section.

CD

CD is a process where the generated executable files or packages (in the CI process) are installed or deployed on application servers in an automated manner. So, CI is basically the first step toward achieving CD.

There is a difference between continuous deployment and delivery, but most of the time, you will be seeing or implementing continuous delivery, especially when you are in the financial sector or any critical business.

Continuous delivery is a process whereby, after all the steps of CI, building and testing then deploying to the application server happens with human intervention. Human intervention means either clicking a button on Build Tools to deploy or allowing a slack bot by approving it. The continuous deployment process differs slightly, whereby the deployment of a successful build to the application server takes place in an automated way and without human intervention.

The processes are illustrated in the following diagram:

Figure 1.2 – CD processes

On reading up to this point, you must be thinking that the CI process is more complex than the CD process, but the CD process is trickier when deploying an application to the production server, especially when it is serving thousands to millions of end-user customers. Any bad experience with the application that is running on your production server may lose customers, which results in a loss for the business. For instance, version 1 (v1) of your application is running live right now and your manager has asked you to deploy version v1.1, but during the deployment, some problem occurred, and somehow v1.1 is not running properly, so you then have to roll back to the previous version, v1. So, all these things now need to be planned and automated in a deployment strategy.

Some CD strategies used in DevOps methodologies are mentioned in the following list:

• Blue-green deployment
• Canary deployment
• Recreate deployment
• A/B testing deployment

Let's have a look at these strategies in brief.

Blue-green deployment

A blue-green deployment is a deployment strategy or pattern where we can reduce downtime by having two production environments. It provides near-zero-downtime rollback capabilities. The basic idea of a blue-green deployment is to shift the traffic from the current environment to another environment. The environments will be identical and run the same application but will have different versions.

You can see an illustration of this strategy in the following diagram:

Figure 1.3 – Blue-green demonstration

In the preceding diagram, we can see that initially, the live application was running in the blue environment, which has App v1; later, it got switched to green, which is running the latest version of the application, App v2.

Canary deployment

In a canary deployment strategy, applications or services get deployed in an incremental manner to a subset of users. Once this subset of users starts using an application or service, then important application metrics are collected and analyzed to decide whether the new version is good to go ahead at full scale to be rolled to all users or needs to roll back for troubleshooting. All infrastructure in production environments is updated in small phases (for example, 10%, 20%, 50%, 75%, 100%).

You can see an illustration of this strategy in the following diagram:

Figure 1.4 – Canary deployment phases

Let's move on to the next strategy.

Recreate deployment

With this deployment strategy, we stop the older version of an application before deploying the newer version. For this deployment, downtime of service is expected, and a full restart cycle is executed.

You can see an illustration of this strategy in the following diagram:

Figure 1.5 – Recreate deployment steps

Let's have a look at the next deployment strategy.

A/B testing deployment

A/B testing is a deployment whereby we run different versions of the same application/services simultaneously, for experimental purposes, in the same environment for a certain period. This strategy consists of routing the traffic of a subset of users to a new feature or function, then getting their feedback and metrics, and after that comparing this with the older version. After comparing the feedback, the decision-maker will update the entire environment with the chosen version of the application/services.

You can see an illustration of this strategy in the following diagram:

Figure 1.6 – A/B testing demonstration

So far, we got to see the deployment strategies, but we do not deploy the application in the production server just after having the build artifact ready from the CI process. We deploy and test the application in various environments and, post success, we deploy in the production environment. We will now see how application versions relate to branches and environments.

Branching strategy (Gitflow)

In the preceding two sections, we got to know about CI and CD, but it is not possible to have a good CI and CD strategy if you do not have a good branching strategy. However, what does branching strategy mean and what exactly are branches?

Whenever a developer writes code in a local machine, after completing the code, they upload/push it to a VCS (Git). The reason for using a VCS is to store the code so that it can be used by other developers and can be tracked and versioned. When a developer pushes the code to Git for the first time, it goes to the master/main branch. A branch in Git is an independent line of development and it serves as an abstraction for the edit/commit/stage process. We will explore the Gitflow branching strategy with a simple example.

Suppose we have a project to implement a calculator. The calculator will have functions of addition, subtraction, multiplication, division, and average. We have three developers (Lily, Brice, and Geraldine) and a manager to complete this project. The manager has asked them to deliver the addition function first. Lily quickly developed the code, built and tested it, pushed it to the Git main/master branch, and tagged it with version 0.1, as illustrated in the following screenshot. The code in the main/master Git branch always reflects the production-ready state, meaning the code for addition will be running in the production environment.

Figure 1.7 – Master branch with the latest version of the code

Now, the manager has asked Brice to start the development of subtraction and multiplication functions as the major functionality for a calculator project for the next release and asked Geraldine to develop division and average functions as a functionality for a future release. Thus, the best way to move ahead is to create a develop branch that will be an exact replica of the working code placed in the master branch. A representation of this branch can be seen in the following screenshot:

Figure 1.8 – Checking out the develop branch from master

So, once a develop branch gets created out of the master, it will have the latest code of the addition function. Now, since Brice and Geraldine must work on their task, they will create a feature branch out of the develop branch. Feature branches are used by developers to develop new features for upcoming releases. It branches off from the develop branch and must merge into the develop branch back once the development of the new functionality completes, as illustrated in the following screenshot:

Figure 1.9 – Creating feature branches from the develop branch

While Brice (responsible for subtraction and multiplication) and Geraldine (responsible for division and average) have been working on their functionality and committing their branches, the manager has found a bug in the current live production environment and has asked Lily to fix that bug. It is never a good practice and is not at all recommended to fix any bug in a production environment. So, what Lily will have to do is to create a hotfix branch from the master branch, fix the bug in code, then merge it into the master branch as well as the develop branch. Hotfix branches are required to take immediate action on the undesired status of the master branch. It must branch off from the master branch and, after fixing the bug, must merge into the master branch as well as the develop branch so that the current develop branch does not have that bug and can deploy smoothly in the next release cycle. Once the fixed code gets merged into the master branch, it gets a new minor version tag and is deployed to the production environment.

This process is illustrated in the following screenshot:

Figure 1.10 – Checking out hotfix from master and later merging into the master and develop branches

Now, once Brice completes his development (subtraction and multiplication), he will then merge his code from the feature branch into the develop branch. But before he merges, he needs to raise a PR/MR. As the name implies, this requests the maintainer of the project to merge the new feature into the develop branch, after reviewing the code. All companies have their own requirements and policies enforced before a merge. Some basic requirements to get a feature merged into the develop branch are that the feature branch should get built successfully without any failures and must have passed a code quality scan.

You can see an example of a PR/MR in the following diagram:

Figure 1.11 – PR raised from the feature branch, post-approval, merged into the develop branch

Once Brice's code (subtraction and multiplication feature) is accepted and merged into the develop branch, then the release process will take place. In the release process, the develop branch code gets merged to the release branch. The release branch basically supports preparation for a new production release. The code in the release branch gets deployed to an environment that is similar to a production environment. That environment is known as staging (pre-production). A staging environment not only tests the application functionality but also tests the load on the server in case traffic increases. If any bugs are found during the test, then these bugs need to be fixed in the release branch itself and merged back into the develop branch.

The process is illustrated in the following screenshot:

Figure 1.12 – Merging the code into release branch from develop branch, fixing bugs (if any), and then merging back into develop branch

Once all the bugs are fixed and testing is successful, the release branch code will get merged into the master branch, then tagged and deployed to the production environment. So, after that, the application will have three functions: addition, subtraction, and multiplication. A similar process will take place for the new features of division and average developed by Geraldine, and finally, the version will be tagged as 1.1 and deployed in the production environment, as illustrated in the following diagram:

Figure 1.13 – Merging code from the release branch into the master branch

These are the main branches in the whole life cycle of an application:

• Master
• Develop

But during a development cycle, the supporting branches, which do not persist once the merge finishes, are shown here:

• Feature
• Hotfix
• Release

Since we now understand branching and CI/CD, let's club all the pieces together, as follows:

Figure 1.14 – CI/CD stages

So, in the preceding diagram, we can see that when a developer finishes their work in a feature branch, they try to raise a PR to the develop branch and the CI pipeline gets triggered. The CI pipeline is nothing but an automated flow or process in any CI tool such as Jenkins/AWS CodePipeline. This CI pipeline will validate whether the feature branch meets all the criteria to get merged into the develop branch. If the CI pipeline runs and build successfully, then the lead maintainer of the project will merge the feature branch into the develop branch, where another automated CI pipeline will trigger and try to deploy the new feature in the development environment. This whole process is known as CI (colored in blue in the preceding diagram). Post-deployment in the development environment, some automated test runs on top of it. If everything goes well and all the metrics look good, then the develop branch gets merged into the staging branch. During this merge process, another automated pipeline gets triggered, which deploys the artifact (uploaded during the develop branch CI process) into the staging environment. The staging environment is generally a close replica of the production environment, where some other tests such as Dynamic Application Security Testing (DAST) and load stress testing take place. If all the metrics and data from the staging environment look good, then staging of the branch code gets merged into the master branch and tagged as a new version.

If the maintainer deploys the tagged artifact in the production environment, then it is considered as continuous delivery. If the deployment happens without any intervention, then it is considered as continuous deployment.

So far, we have learned how application development and deployment take place. The preceding concept of CI/CD and branching strategies were quite important to be familiar with to move ahead and understand the rest of the chapters. In the next section, we will be learning about the AWS-managed CI/CD service CodeStar and will use it to create and deploy a project in development, staging, and production environments.