This chapter is intended to reflect the recent development and evolution of the software delivery approach, mostly focusing on changes to CD and connected advancements underpinned by the emergence of CI technologies, new tools, processes, practices, and management structures. To understand the advancements of CD in the cloud better, it is important to connect the dots and learn about internal and external factors influencing the advancement of CD. The term CD has been around for a long time; yet, let us start from the basics. The most widely referred definition of CD is provided by Jez Humble (and can be found at https://continuousdelivery.com/):

“Continuous Delivery is the ability to get the changes of all types of new features, config changes, bug fixes, and experiments into production or into the hands of the user safely and quickly in a sustainable way.”

With advancements in CD, there are several attempts to redefine or refine the definition of CD. For example, AWS (at https://aws.amazon.com/devops/continuous-delivery/) defines CD as follows:

“Continuous delivery is a software development practice where code changes are automatically prepared for a release to production.”

According to another definition by Azure (at https://docs.microsoft.com/en-us/devops/deliver/what-is-continuous-delivery), CD is defined as follows:

“Continuous delivery (CD) is the process of automating build, test, configuration, and deployment from a build to a production environment.”

To understand it from Google’s perspective (at https://cloud.google.com/solutions/continuous-delivery), it is simply the following:

“End-to-end automation from source to production.”

Although there are variances and differences in definitions of CD, it makes it easier for us to reflect that CD is evolving and is defined by a set of characteristics that work together in the ecosystem.

As we move ahead in this book, we will describe the building blocks of CD and their relationship.

Defining the CD ecosystem

For now, let’s try to understand the factors influencing the evolution of CD to define the ecosystem:

• Speed: It is evident that most of the definitions of CD lead to automation, making it visible that speed is one of the core drivers for the evolution of the CD and its connected ecosystem. Many key advancements are linked to removing the bottleneck to speed and lead time: starting from the evolution of practices such as CI/CD to the explosion of cloud-based pre-built pipelines, as well as including automation in build-test-deploy practices.
• Security: The fundamental shift toward securing software and addressing key risks with the utmost priority has led to the systematic evolution of the CD ecosystem with new tools and techniques, including security from the inception stages of CD.
• Software development itself: Developers are inclined to move away from traditional approaches to software development to more modern ways with the integration of Agile methodology, making use of more progressive, efficient tools and processes to eliminate lead times and quality issues. One example of such an application is Azure Boards, which provides a choice of Agile planning tools, many of which work in combination with CD practices.
• Software life cycle management: With velocity, scale, interdependence, and the growing complexity of the software landscape comes the need to simplify software life cycle management. Quick discovery and fixing of software bugs, accelerating software updates, comprehensive assessment of dependencies, and razor-sharp focus on vulnerability management are essential elements.
• Software operations: Finally, the return on investment (ROI) on software is realized in production. DevOps, SRE, and the emergence of cloud-based services are revolutionizing our CD ecosystem with a focus on building cross-functional teams, platforms, and tools.

The preceding factors indicate that the CD ecosystem comprises dynamic internal and external components. Taking inspiration from a natural ecosystem, if we attempt to define the CD ecosystem, we come up with this:

The CD ecosystem is a distributed, adaptive, and techno-functional system of CD practices, processes, tools, and techniques that support the evolution of CD.

Characteristics of the CD ecosystem

In order to be able to deliver continuously, we need a system of practices, processes, tools, and techniques that are reliable and can produce repeatable results consistently. Some of the basic ingredients for providing such outcomes are the following:

• Self-organizing and adaptive: The CD ecosystem, as with any other such system, must be in a state of equilibrium. No single specific outcome can drive the construct and choice of practices, tools, or techniques entirely. For example, speed is one driver for CD, but the ecosystem must have components that keep the ecosystem close to equilibrium by including components/tools through which dynamics of security, speed, quality, and value are presented in a balanced way.
• Dynamic: The CD ecosystem consists of evolving tools and practices and integrates with emerging technology, so it makes sense that the CD ecosystem is dynamic. For example, the evolution of Kubernetes from an internal container ecosystem at Google managed by a community of contributors into an enterprise-grade de facto orchestration engine, with its adoption by cloud providers, adds an array of new features to CD. Interoperability and event-driven architecture evolution are key areas of progress to emphasize the dynamic characteristic of the CD ecosystem.
• Distributed: The CD ecosystem should be able to provide our developers and practitioners with an array of tools and services. It connects directly to the evolution of the marketplace approach. For example, cloud service providers such as Azure enable you to build and deploy applications on an array of platforms and languages. Azure also offers two options for version control—Git and Azure Repos, and many more such choices.
• Supports an “as-a-service” model: With time, the key components of the CD ecosystem will be degraded or replaced by other new and advanced features. We must focus on a consumption-based model for features and measure their usage over time. One of the key drivers of an as-a-service model is that it helps to build a self-regulating and self-sustaining ecosystem. Many services from cloud-based CI/CD tools and application vendors have community versions and tiered subscriptions for on-premises, hybrid, or enterprise versions, supporting a pay-per-use pricing model.

Key components of the CD ecosystem

For any ecosystem to thrive, the key components must act together to advance it by catering to evolving demands. The CD ecosystem will continue to evolve if progressive tools and applications continue to challenge the status quo – right from basic CI/CD tools and applications for version control and workflow automation through to specialized cloud-native CI/CD tools that can be run on any cloud and build a strong progressive foundation anywhere.

The CD ecosystem comprises three key components that continuously steer the evolution of the ecosystem by delivering value within and outside the boundaries of the ecosystem CI/CD tools and applications: consumers, producers, and practitioners. Let us look at them in detail:

• The role of CI/CD tools and applications as a consumer: The increasing adoption of digital technologies creates huge market potential for CD. If CD provides a competitive advantage to organizations by delivering software features frequently reliably and cost-effectively, the ecosystem is bound to flourish. The role of consumers in the ecosystem is to provide feedback to CI/CD tools and application vendors and continue to experiment and modernize the CI/CD pipeline with progressive features, tools, and applications to help unlock new revenue streams.
• The role of CI/CD tools and applications as producers: The CI/CD ecosystem needs investment and integration with emerging technologies to facilitate a competitive advantage for consumers. The tools and applications that are the foundation of the CD ecosystem should also be the consumers of the ecosystem. For example, CI/CD tools and applications are often architected with microservices and rely on container-based deployment. They are designed to support interoperability and integrate well with other cloud-native tools and cloud-based services.
• The role of CI/CD practitioners as an actor in an ecosystem: The practitioners are the final actors in the ecosystem through which value is realized. The practitioners will churn out the waste in the CI/CD pipeline. Through the adoption of various tools and applications, practitioners will continuously optimize the ecosystem, and create perspectives on new applications, tools, and features. For example, pushing toward more interoperability standards with the rise of hybrid pipelines, the adoption of co-pilot features, and a more platform-centric approach toward CD to deliver software features seamlessly is paving the way for new developments.

Managing the CD ecosystem

Several organizations have already embarked on their CD journeys. With that, the need to effectively manage investments, develop and improve the current technical adoption, and steer strategic outcomes through CD becomes an urgent priority.

Many industry reports highlight that the full potential of CI/CD and the connected ecosystem is yet to be realized. There are multiple reasons associated with this, such as siloed initiatives, ad hoc investment, and lack of upskilling. These factors indicate the need to manage the CI/CD landscape with a multidimensional outlook.

The management of CD also must be structured yet agile in a way. Planning, organizing, implementing, and monitoring the outcome of CD must be modern and fit for purpose. To avoid delays, bottlenecks, and constraints, the management process should be tailored to CD.

In this subsection, we will present some guidance on effectively managing CD. We will start with a CD reference architecture, further elaborate on the role of CD frameworks, and eventually highlight the need to have a CD Community of Practices (CoPs). In subsequent chapters, we will detail these aspects along with other important dimensions of creating and implementing CD in the cloud strategically:

• CD reference architecture: This refers to the abstract concepts that can outline a structure or a construct for CD:
  • Provides a common vocabulary
  • Provides guidance about the functions and their interaction (e.g., APIs)
  • Can be defined at a different level of abstraction
  • Can be instantiated for a particular domain or specific projects
  • Can be mapped to specific sets of goals, as no single architecture can fulfill all needs

An example of a reference architecture and associated service offering in the cloud is Open Source on AWS: https://aws.amazon.com/blogs/devops/choosing-a-well-architected-ci-cd-approach-open-source-on-aws/.

Another example of a CD reference architecture is one being worked on by the Continuous Delivery Foundation (CDF):

• A CD framework: A framework is a pre-built general or special-purpose architecture that’s designed to be extended for a set of consumers, such as the healthcare industry, the defense sector, or the financial community. CD tools, along with cloud technologies, are constantly undergoing changes to improve the software development environment and practices. It is hard to keep pace with upskilling without structured support from the community of practitioners.

As an example of a community of practice, here are a collection of technical articles and blogs published or curated by Google Cloud Developer Advocates: https://medium.com/google-cloud.

The AWS Community Builders program is one of the unique programs launched by AWS for the community of practitioners to share resources and educational content and build a networking ecosystem to support the community.

Other communities provide similar guidance: the DevOps Institute’s DevOps in the Wild community and the CDF community for open source CD tools and best practices, to name two.

So far, we have introduced some terminology we will use throughout the book, and this section has ensured that you have a good understanding of the main concepts of CD and its key components. We also touched upon the aspects of CD in the cloud, which will be illustrated in the following chapters in detail.

In this section, we will look at the benefits of CD in more detail. Building upon the idea of the key characteristics of CD being speed, enabling security and progressive practices for software development, and operation and life cycle management, let’s look at the benefits.

There are some industry references. DevOps Research And Assessment (DORA) is an initiative by Google to support organizations to achieve high performance by embracing DevOps according to actionable guidance, DORA metrics (https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance), which can give you a measure of the positive business outcomes from adopting CD. For us, it will be easier to map the key characteristics to the business outcomes and key benefits of CD:

Table 1.1 — Mapping CD characteristics to key benefits

Let us now look at the benefits of CD in the cloud.

Benefits of CD in the cloud

Taking one step forward, cloud service providers combined the benefits with the service offerings to facilitate better business outcomes:

Table 1.2 — Mapping CD’s benefits to cloud service provider offering

In this section, you read a short overview of the CD ecosystem and how it can help you in achieving your goals. In the next sections, you will be introduced to the differences between CI and CD and where CD in the cloud can support you.

When delivering software to our customers, the customer assumes that it runs reliably and without significant problems. Any misfunction or outage of the software—depending on its purpose—might lead to a loss of reputation on the customer side and the people and company that developed the software. Think of an issue where code was written weeks or months ago, the software is already deployed in the customer’s environment, and it crashes randomly after some time. While troubleshooting, we might tackle the following issues:

• Time pressure: As the software has already been delivered to production and customers have been affected by its misbehavior, it is obvious to provide a resolution during this process. As people providing this might work under pressure, it’s not unlikely that new issues might be introduced.
• Determine which change introduced the misbehavior: In our case, the change might have been introduced a long time ago, so one of our major challenges might be finding out when or in which context the change has been introduced.
• Finding the needle in the haystack: Our software might consist of lots of components. If we can’t find out which change introduced the issue, we might want to find where in the code it happens.
• Dev/prod parity: As our working copy of the code might already have progressed further than the version we are fixing the issue for, we will also have to update newer versions of the software.

These are only examples of problems we might face while fixing such issues. To tackle such cases, we want to find problems in our software very early and build automation around building, testing, and publishing artifacts, called CI. The process of CI should ensure that the code checked in is continuously built and tested. Therefore, problems are found at a very early stage.

Using CI and CD methods, we can find problems early and avoid them hitting the customer’s system. Before we dive deeper into CD processes and implementation, we will take a brief look at the building blocks of a typical CI/CD infrastructure:

Figure 1.1 – Building blocks of a CD infrastructure

In current development setups, developers store their code in source code management (SCM) systems, such as Git, Mercurial, or Subversion. CI systems, such as Jenkins or Tekton, can either watch these repositories for changes on specific branches or tags or get triggered by the SCM systems to initiate workflows. These workflows may include static code analysis steps, such as linting and static security checks, before building, but also more dynamic tests, which can only be done against the running system, such as API tests or Dynamic Application Security tests.

Usually, these steps run on every commit in so-called feature branches. At some point in time, a developer might decide that a developed code part (feature) might be finished and therefore include it in the shared code base and raise a pull or merge request. As a best practice, there might be code reviews for this integration step, which can be automatic or manual. Humans will do the manual part. Furthermore, automatic checks can be done by a CI workflow, which should ensure that the code checked into the code repository is not only compilable but also stable and, in the best case, secure.

Depending on the release strategy, a successful merge to a central shared repository could trigger the CD process. This could also be done when the code is merged into specific branches or with defined tags. As discussed in later chapters of this book, the code will be deployed in various environments (stages) during the further CD process, and tests should ensure that the software is in a deployable state at the end of the process, and that deliverable artifacts are available. Although the terms CI and CD are often used interchangeably, the process of deploying automatically to the production systems is called CD.

In this section, we learned that it is important to find problems in software early by building and testing software continuously. Furthermore, CD helps us keep the software in a deployable state at any time. Last but not least, CD can help us in accelerating our time to market by deploying to production automatically.

Many issues could come up with continuously deploying to production. Even if the software is tested well during the previous phases and our deployment mechanisms are working perfectly, we can encounter situations where everything is running from a technical perspective and still the experience for the end users is very dissatisfying. For instance, a customer might click on a button on a website, and it takes about 30 seconds to get a response, although there is not much load on the system. Such things could come up because the software is not tested under the conditions present in the production environment, and the things users are executing have not been tested in this constellation before. Long story short, we should always aim to deploy our software often and automatically, but we need additional mechanisms that safeguard our deployment process. One of them is progressive delivery.

“Progressive delivery is the process of pushing changes to a product iteratively, first to a small audience and then to increasingly larger audiences to maintain quality control.”

(Orit Golowinski, https://www.devopsinstitute.com/progressive-delivery-7-methods/)

Using progressive delivery, the software is deployed in a phased way, ensuring that not all users are affected by a problem. In Chapter 6, we will take a closer look at Blue-Green Deployments and Canary Releases, which can be used to not only create preview environments but also to shift production traffic between different versions of a product. At this point, we are able to deploy automatically after a new feature has been introduced and might be able to deliver it to a small user base to see how it performs. As we aim for automation and might not want to make such decisions manually, modern progressive delivery controllers are able to make such decisions based on data. For instance, a new software version gets automatically deployed to 1 of 5 production environments. We want to ensure that requests do not take longer than 500 milliseconds, and in the newer release, the response time must not be more than 10% higher than before. Therefore, we monitor such parameters and configure our progressive delivery controller to wait for some time and shift the next 20% of traffic if our conditions are met.

Progressive delivery is not only used to limit the blast radius of errors but also to validate user acceptance of newly implemented features. Using observability data, we could also find out whether there is a decrease in orders in various timeframes when a new feature is rolled out and can automatically roll back the feature in this case.

We will dive deeper into the various deployment strategies, as well as observability and feature flagging, in Chapter 6 and Chapter 7.

All of the things we discussed in the previous sections could be implemented in your own data center. The code could be hosted in a repository, a local CI server might watch for changes, and CD tooling could deploy the applications to the production environments. In the rest of the book, however, we will mainly cover CD in the cloud. Therefore, we assume that the software is built, deployed, and operated in cloud environments. We will go more into detail on cloud characteristics and delivery models in Chapter 2. In this section, we will deal with a few technologies, as well as the benefits and drawbacks when doing CD in the cloud.

When dealing with CD and deployment, we will often refer to cloud-native applications and technologies, which are defined as follows.

Cloud-native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil (the Cloud Native Computing Foundation – https://www.cncf.io/about/who-we-are/).

As defined previously, many technologies and products may be involved when delivering cloud-native systems. We will often hear three terms: virtualization, containers, and container orchestration/Kubernetes. Let us look at them in some detail:

• Virtualization is one of the core techniques in cloud computing and allows more efficient usage of resources and supports the implementation of typical cloud characteristics; for example, on-demand self-service, resource pooling, and rapid elasticity. Using virtualization allows the abstraction of software from hardware; typically, virtualization is referred to the usage of virtual machines, where compute resources of one physical host are partitioned into multiple virtual machines.
• Containers provide a way to package the runtime environment, as well as the application code, into a single artifact that can run on a container runtime. While containers have existed for a long time in the industry (OpenVZ and BSD jails), Docker extended the technology to build containers in a systematic way and to easily add containerization to the development process. While virtual machines introduce some overhead into the system, as they mostly run their own kernel, containers run with almost no overhead, which leads to much better resource usage and a higher possible load of workloads on one machine.
• Container orchestration deals with many containers in many physical environments and ensures that container workloads are not only scheduled on the right node but also get the resources they need. At the time of writing, Kubernetes is the de facto standard for container orchestration. One of the main benefits of using a container orchestration platform is the simple fault tolerance and scaling of applications. The scheduler of the platform takes care of the desired number of containers and the user’s needs, and—if the resources are available—manages them according to the configured specifications. Furthermore, the usage of such an infrastructure enforces the operators to use configurations, which may either be in the format provided by the platform or in a framework for managing deployments for this platform. As a result, the used infrastructure is documented in code and can be recreated and duplicated with little effort. Orchestration platforms generally provide load-balancing mechanisms and the possibility to define health and readiness checks, which makes it easy for users to build auto-scaling applications.

When deploying to cloud-based systems, we also must think about the infrastructure. To define the infrastructure, the term Infrastructure as Code has emerged, which is the practice of describing the target state of the infrastructure in declarative language. One of the major challenges we will also cover in this book is the convergence of the infrastructure and the application deployment.

Every major cloud service provider provides a framework for deploying applications automatically to their environment. There are Infrastructure-as-Code modules available for various toolsets and proprietary solutions. As a further consequence, each cloud provider provides a managed code repository and CI tooling to build software from code and container registries to store the container images. Last but not least, there are many possibilities for running containers/applications in the cloud providers, such as a managed Kubernetes service:

Table 1.3 — Examples of cloud services for CD

In this section, we discussed a few basics of cloud-based CD implementation and services, which can be consumed through cloud providers. As we progress through this book, we will use the resources provided by these cloud providers, as well as tools that can be installed on cloud provider resources.

In this chapter, we introduced the foundational aspects of CD, starting with its definition and characteristics, and explaining the CD ecosystem at a high level.

Once we learned about all the benefits of CD, we took a closer look at the technical side and its prerequisites and learned that CI helps us build and test software continuously. Furthermore, CD ensures that our software is in a deployable state at any time, and Continuous Deployment, which is often used interchangeably with CD automates the deployment of our application to production systems.

As failures in automated deployments could have a very large blast radius, progressive delivery can reduce this by gradually deploying the application to a limited number of users. To also automate the evaluation of the behavior of the systems, we might rely on data from our observability solution to make decisions if the criteria to shift more traffic to the new version are met. Last but not least, we can also use these mechanisms to assess the business impact of a new feature and take action if this leads to, for example, a loss of revenue.

In the final part of this chapter, we started to deal with cloud-specifics related to CD that we will need in future chapters of the book. Therefore, we introduced and explained the terms Virtualization, Containers, and Container Orchestration and learned that we need to take care of our infrastructure (using Infrastructure as Code) before deploying applications on it. Finally, we discovered that there are many cloud services that will help us on our journey to CD in the cloud.

Throughout the next chapter, we will go deeper into the characteristics of the cloud and its delivery models. Furthermore, you will learn why delivering to the cloud is different from traditional environments.

• The book Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation by Jez Humble and David Farley