What is Continuous Delivery?

The most accurate definition of the Continuous Delivery is stated by Jez Humble and reads as follows: "Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes, and experiments—into production, or into the hands of users, safely and quickly in a sustainable way." That definition covers the key points.

To understand it better, let's imagine a scenario. You are responsible for the product, let's say, the email client application. Users come to you with a new requirement—they want to sort emails by size. You decide that the development will take around one week. When can the user expect to use the feature? Usually, after the development is done, you hand over the completed feature first to the QA team and then to the operations team, which takes additional time ranging from days to months. Therefore, even though the development took only one week, the user receives it in a couple of months! The Continuous Delivery approach addresses that issue by automating manual tasks so that the user could receive a new feature as soon as it's implemented.

To present better what to automate and how, let's start by describing the delivery process that is currently used for most software systems.

The traditional delivery process

The traditional delivery process, as the name suggests, has been in place for many years now and is implemented in most IT companies. Let's define how it works and comment on its shortcomings.

Introducing the traditional delivery process

Any delivery process begins with the requirements defined by a customer and ends up with the release on the production. The differences are in between. Traditionally, it looks as presented in the following release cycle diagram:

The release cycle starts with the requirements provided by the Product Owner, who represents the Customer (stakeholders). Then there are three phases, during which the work is passed between different teams:

• Development: Here, the developers (sometimes together with business analysts) work on the product. They often use Agile techniques (Scrum or Kanban) to increase the development velocity and to improve the communication with the client. Demo sessions are organized to obtain a customer's quick feedback. All good development techniques (like test-driven development or extreme programming practices) are welcome. After the implementation is completed, the code is passed to the QA team.
• Quality Assurance: This phase is usually called User Acceptance Testing (UAT) and it requires a code freeze on the trunk codebase, so that no new development would break the tests. The QA team performs a suite of Integration Testing, Acceptance Testing, and Non-functional Testing (performance, recovery, security, and so on). Any bug that is detected goes back to the development team, so developers usually also have their hands full of work. After the UAT phase is completed, the QA team approves the features that are planned for the next release.
• Operations: The last phase, usually the shortest one, means passing the code to the Operations team, so that they can perform the release and monitor the production. If anything goes wrong, they contact developers to help with the production system.

The length of the release cycle depends on the system and the organization, but it usually ranges from a week to a few months. The longest I've heard about was one year.The longest I worked with was quarterly-based and each part took as follows: development-1.5 months, UAT-1 month and 3 weeks, release (and strict production monitoring)-1 week.

The traditional delivery process is widely used in the IT industry and it's probably not the first time you've read about such an approach. Nevertheless, it has a number of drawbacks. Let's look at them explicity to understand why we need to strive for something better.

Shortcomings of the traditional delivery process

The most significant shortcomings of the traditional delivery process include the following:

• Slow delivery: Here, the customer receives the product long after the requirements were specified. It results in the unsatisfactory time to market and delays of the customer's feedback.
• Long feedback cycle: The feedback cycle is not only related to customers, but also to developers. Imagine that you accidentally created a bug and you learn about it during the UAT phase. How long does it take to fix something you worked on two months ago? Even minor bugs can consume weeks.
• Lack of automation: Rare releases don't encourage the automation, which leads to unpredictable releases.
• Risky hotfixes: Hotfixes can't usually wait for the full UAT phase, so they tend to be tested differently (the UAT phase is shortened) or not tested at all.
• Stress: Unpredictable releases are stressful for the operations team. What's more, the release cycle is usually tightly scheduled which imposes an additional stress on developers and testers.
• Poor communication: Work passed from one team to another represents the waterfall approach, in which people start to care only about their part, rather than the complete product. In case anything goes wrong, that usually leads to the blaming game instead of cooperation.
• Shared responsibility: No team takes the responsibility for the product from A to Z. For developers: "done" means that requirements are implemented. For testers: "done" means that the code is tested. For operations: "done" means that the code is released.

• Lower job satisfaction: Each phase is interesting for a different team, but other teams need to support the process. For example, the development phase is interesting for developers but, during two other phases, they still need to fix bugs and support the release, which usually is not interesting for them at all.

These drawbacks represent just a tip of the iceberg of the challenges related to the traditional delivery process. You may already feel that there must be a better way to develop the software and this better way is, obviously, the Continuous Delivery approach.

Benefits of Continuous Delivery

“How long would it take your organization to deploy a change that involves just one single line of code? Do you do this on a repeatable, reliable basis?" These are the famous questions from Mary and Tom Poppendieck (authors of Implementing Lean Software Development), which have been quoted many times by Jez Humble and other authors. Actually, the answer to these questions is the only valid measurement of the health of your delivery process.

To be able to deliver continuously, and not to spend a fortune on the army of operations teams working 24/7, we need automation. That is why, in short, Continuous Delivery is all about changing each phase of the traditional delivery process into a sequence of scripts, called the automated deployment pipeline or the Continuous Delivery pipeline. Then, if no manual steps are required, we can run the process after every code change and, therefore, deliver the product continuously to the users.

Continuous Delivery lets us get rid of the tedious release cycle and, therefore, brings the following benefits:

• Fast delivery: Time to market is significantly reduced as customers can use the product as soon as the development is completed. Remember, the software delivers no revenue until it is in the hands of its users.
• Fast feedback cycle: Imagine you created a bug in the code, which goes into the production the same day. How much time does it take to fix something you worked on the same day? Probably not much. This, together with the quick rollback strategy, is the best way to keep the production stable.
• Low-risk releases: If you release on a daily basis, then the process becomes repeatable and therefore much safer. As the saying goes, "If it hurts, do it more often."
• Flexible release options: In case you need to release immediately, everything is already prepared, so there is no additional time/cost associated with the release decision.

Needless to say, we could achieve all the benefits simply by eliminating all delivery phases and proceeding with the development directly on the production. It would, however, cause the quality to decline. Actually, the whole difficulty of introducing Continuous Delivery is the concern that the quality would decrease together with eliminating manual steps. In this book, we will show how to approach it in a safe manner and explain why, contrary to common beliefs, the products delivered continuously have fewer bugs and are better adjusted to the customer's needs.

Success stories

My favorite story on Continuous Delivery was told by Rolf Russell at one of his talks. It goes as follows. In 2005, Yahoo acquired Flickr and it was a clash of two cultures in the developer's world. Flickr, by that time, was a company with the start-up approach in mind. Yahoo, on the contrary, was a huge corporation with strict rules and the safety-first attitude. Their release processes differed a lot. While Yahoo used the traditional delivery process, Flickr released many times a day. Every change implemented by developers went on the production the same day. They even had a footer at the bottom of their page showing the time of the last release and the avatars of the developers who did the changes.

Yahoo deployed rarely and each release brought a lot of changes well tested and prepared. Flickr worked in very small chunks, each feature was divided into small incremental parts and each part was deployed quickly to the production. The difference is presented in the following diagram:

You can imagine what happened when the developers from two companies met. Yahoo obviously treated Flickr's colleagues as junior irresponsible developers, "a bunch of software cowboys who don't know what they are doing." So, the first thing they wanted to change was to add a QA team and the UAT phase into Flickr's delivery process. Before they applied the change, however, Flickr's developers had only one wish. They asked to evaluate the most reliable products in the whole Yahoo company. What a surprise when it happened that of all the software in Yahoo, Flickr had the lowest downtime. The Yahoo team didn't understand it at first, but let Flickr stay with their current process anyway. After all, they were engineers, so the evaluation result was conclusive. Only after some time, they realized that the Continuous Delivery process can be beneficial for all products in Yahoo and they started to gradually introduce it everywhere.

The most important question of the story remains-how was it possible that Flickr was the most reliable system? Actually, the reason for that fact was what we already mentioned in the previous sections. A release is less risky if:

• The delta of code changes is small
• The process is repeatable

That is why, even though the release itself is a difficult activity, it is much safer when done frequently.

The story of Yahoo and Flickr is only an example of many successful companies for which the Continuous Delivery process proved to be right. Some of them even proudly share details from their systems, as follows:

• Amazon: In 2011, they announced reaching 11.6 seconds (on average) between deployments
• Facebook: In 2013, they announced deployment of code changes twice a day
• HubSpot: In 2013, they announced deployment 300 times a day
• Atlassian: In 2016, they published a survey stating that 65% of their customers practice continuous delivery

You can read more about the research on the Continuous Delivery process and individual case studies at https://continuousdelivery.com/evidence-case-studies/.

Keep in mind that the statistics get better every day. However, even without any numbers, just imagine a world in which every line of code you implement goes safely into the production. Clients can react quickly and adjust their requirements, developers are happy because they don't have to solve that many bugs, managers are satisfied because they always know what is the current state of work. After all, remember, the only true measure of progress is the released software.