Understanding the CD ecosystem

This chapter is intended to reflect the recent development and evolution of the software delivery approach, mostly focusing on changes to CD and connected advancements underpinned by the emergence of CI technologies, new tools, processes, practices, and management structures. To understand the advancements of CD in the cloud better, it is important to connect the dots and learn about internal and external factors influencing the advancement of CD. The term CD has been around for a long time; yet, let us start from the basics. The most widely referred definition of CD is provided by Jez Humble (and can be found at https://continuousdelivery.com/):

“Continuous Delivery is the ability to get the changes of all types of new features, config changes, bug fixes, and experiments into production or into the hands of the user safely and quickly in a sustainable way.”

With advancements in CD, there are several attempts to redefine or refine the definition of CD. For example, AWS (at https://aws.amazon.com/devops/continuous-delivery/) defines CD as follows:

“Continuous delivery is a software development practice where code changes are automatically prepared for a release to production.”

According to another definition by Azure (at https://docs.microsoft.com/en-us/devops/deliver/what-is-continuous-delivery), CD is defined as follows:

“Continuous delivery (CD) is the process of automating build, test, configuration, and deployment from a build to a production environment.”

To understand it from Google’s perspective (at https://cloud.google.com/solutions/continuous-delivery), it is simply the following:

“End-to-end automation from source to production.”

Although there are variances and differences in definitions of CD, it makes it easier for us to reflect that CD is evolving and is defined by a set of characteristics that work together in the ecosystem.

As we move ahead in this book, we will describe the building blocks of CD and their relationship.

Defining the CD ecosystem

For now, let’s try to understand the factors influencing the evolution of CD to define the ecosystem:

• Speed: It is evident that most of the definitions of CD lead to automation, making it visible that speed is one of the core drivers for the evolution of the CD and its connected ecosystem. Many key advancements are linked to removing the bottleneck to speed and lead time: starting from the evolution of practices such as CI/CD to the explosion of cloud-based pre-built pipelines, as well as including automation in build-test-deploy practices.
• Security: The fundamental shift toward securing software and addressing key risks with the utmost priority has led to the systematic evolution of the CD ecosystem with new tools and techniques, including security from the inception stages of CD.
• Software development itself: Developers are inclined to move away from traditional approaches to software development to more modern ways with the integration of Agile methodology, making use of more progressive, efficient tools and processes to eliminate lead times and quality issues. One example of such an application is Azure Boards, which provides a choice of Agile planning tools, many of which work in combination with CD practices.
• Software life cycle management: With velocity, scale, interdependence, and the growing complexity of the software landscape comes the need to simplify software life cycle management. Quick discovery and fixing of software bugs, accelerating software updates, comprehensive assessment of dependencies, and razor-sharp focus on vulnerability management are essential elements.
• Software operations: Finally, the return on investment (ROI) on software is realized in production. DevOps, SRE, and the emergence of cloud-based services are revolutionizing our CD ecosystem with a focus on building cross-functional teams, platforms, and tools.

The preceding factors indicate that the CD ecosystem comprises dynamic internal and external components. Taking inspiration from a natural ecosystem, if we attempt to define the CD ecosystem, we come up with this:

The CD ecosystem is a distributed, adaptive, and techno-functional system of CD practices, processes, tools, and techniques that support the evolution of CD.

Characteristics of the CD ecosystem

In order to be able to deliver continuously, we need a system of practices, processes, tools, and techniques that are reliable and can produce repeatable results consistently. Some of the basic ingredients for providing such outcomes are the following:

• Self-organizing and adaptive: The CD ecosystem, as with any other such system, must be in a state of equilibrium. No single specific outcome can drive the construct and choice of practices, tools, or techniques entirely. For example, speed is one driver for CD, but the ecosystem must have components that keep the ecosystem close to equilibrium by including components/tools through which dynamics of security, speed, quality, and value are presented in a balanced way.
• Dynamic: The CD ecosystem consists of evolving tools and practices and integrates with emerging technology, so it makes sense that the CD ecosystem is dynamic. For example, the evolution of Kubernetes from an internal container ecosystem at Google managed by a community of contributors into an enterprise-grade de facto orchestration engine, with its adoption by cloud providers, adds an array of new features to CD. Interoperability and event-driven architecture evolution are key areas of progress to emphasize the dynamic characteristic of the CD ecosystem.
• Distributed: The CD ecosystem should be able to provide our developers and practitioners with an array of tools and services. It connects directly to the evolution of the marketplace approach. For example, cloud service providers such as Azure enable you to build and deploy applications on an array of platforms and languages. Azure also offers two options for version control—Git and Azure Repos, and many more such choices.
• Supports an “as-a-service” model: With time, the key components of the CD ecosystem will be degraded or replaced by other new and advanced features. We must focus on a consumption-based model for features and measure their usage over time. One of the key drivers of an as-a-service model is that it helps to build a self-regulating and self-sustaining ecosystem. Many services from cloud-based CI/CD tools and application vendors have community versions and tiered subscriptions for on-premises, hybrid, or enterprise versions, supporting a pay-per-use pricing model.

Key components of the CD ecosystem

For any ecosystem to thrive, the key components must act together to advance it by catering to evolving demands. The CD ecosystem will continue to evolve if progressive tools and applications continue to challenge the status quo – right from basic CI/CD tools and applications for version control and workflow automation through to specialized cloud-native CI/CD tools that can be run on any cloud and build a strong progressive foundation anywhere.

The CD ecosystem comprises three key components that continuously steer the evolution of the ecosystem by delivering value within and outside the boundaries of the ecosystem CI/CD tools and applications: consumers, producers, and practitioners. Let us look at them in detail:

• The role of CI/CD tools and applications as a consumer: The increasing adoption of digital technologies creates huge market potential for CD. If CD provides a competitive advantage to organizations by delivering software features frequently reliably and cost-effectively, the ecosystem is bound to flourish. The role of consumers in the ecosystem is to provide feedback to CI/CD tools and application vendors and continue to experiment and modernize the CI/CD pipeline with progressive features, tools, and applications to help unlock new revenue streams.
• The role of CI/CD tools and applications as producers: The CI/CD ecosystem needs investment and integration with emerging technologies to facilitate a competitive advantage for consumers. The tools and applications that are the foundation of the CD ecosystem should also be the consumers of the ecosystem. For example, CI/CD tools and applications are often architected with microservices and rely on container-based deployment. They are designed to support interoperability and integrate well with other cloud-native tools and cloud-based services.
• The role of CI/CD practitioners as an actor in an ecosystem: The practitioners are the final actors in the ecosystem through which value is realized. The practitioners will churn out the waste in the CI/CD pipeline. Through the adoption of various tools and applications, practitioners will continuously optimize the ecosystem, and create perspectives on new applications, tools, and features. For example, pushing toward more interoperability standards with the rise of hybrid pipelines, the adoption of co-pilot features, and a more platform-centric approach toward CD to deliver software features seamlessly is paving the way for new developments.

Managing the CD ecosystem

Several organizations have already embarked on their CD journeys. With that, the need to effectively manage investments, develop and improve the current technical adoption, and steer strategic outcomes through CD becomes an urgent priority.

Many industry reports highlight that the full potential of CI/CD and the connected ecosystem is yet to be realized. There are multiple reasons associated with this, such as siloed initiatives, ad hoc investment, and lack of upskilling. These factors indicate the need to manage the CI/CD landscape with a multidimensional outlook.

The management of CD also must be structured yet agile in a way. Planning, organizing, implementing, and monitoring the outcome of CD must be modern and fit for purpose. To avoid delays, bottlenecks, and constraints, the management process should be tailored to CD.

In this subsection, we will present some guidance on effectively managing CD. We will start with a CD reference architecture, further elaborate on the role of CD frameworks, and eventually highlight the need to have a CD Community of Practices (CoPs). In subsequent chapters, we will detail these aspects along with other important dimensions of creating and implementing CD in the cloud strategically:

• CD reference architecture: This refers to the abstract concepts that can outline a structure or a construct for CD:
  • Provides a common vocabulary
  • Provides guidance about the functions and their interaction (e.g., APIs)
  • Can be defined at a different level of abstraction
  • Can be instantiated for a particular domain or specific projects
  • Can be mapped to specific sets of goals, as no single architecture can fulfill all needs

An example of a reference architecture and associated service offering in the cloud is Open Source on AWS: https://aws.amazon.com/blogs/devops/choosing-a-well-architected-ci-cd-approach-open-source-on-aws/.

Another example of a CD reference architecture is one being worked on by the Continuous Delivery Foundation (CDF):

• A CD framework: A framework is a pre-built general or special-purpose architecture that’s designed to be extended for a set of consumers, such as the healthcare industry, the defense sector, or the financial community. CD tools, along with cloud technologies, are constantly undergoing changes to improve the software development environment and practices. It is hard to keep pace with upskilling without structured support from the community of practitioners.

As an example of a community of practice, here are a collection of technical articles and blogs published or curated by Google Cloud Developer Advocates: https://medium.com/google-cloud.

The AWS Community Builders program is one of the unique programs launched by AWS for the community of practitioners to share resources and educational content and build a networking ecosystem to support the community.

Other communities provide similar guidance: the DevOps Institute’s DevOps in the Wild community and the CDF community for open source CD tools and best practices, to name two.

So far, we have introduced some terminology we will use throughout the book, and this section has ensured that you have a good understanding of the main concepts of CD and its key components. We also touched upon the aspects of CD in the cloud, which will be illustrated in the following chapters in detail.