ACME started out with a couple of things in common with the many thousands of small software businesses scattered around the globe: it had some good ideas and a saw gap in the market that it could exploit to make its fortune. It had a relatively small amount of cash so it needed to move fast to be able to survive and it needed to quickly entice, enlist, and retain customers at all costs. It did this by delivering what the customer wants just before the customer needs it. Deliver too soon, and it may have wasted money on building solutions that the customer decides they no longer want. Deliver too late, and someone else may well have taken the company's market share—and the revenue—away. The keyword here is deliver.

As a small start-up, in the early days, the going is slow and the work is hard: lots of R&D, frantically-built pre-sales prototypes, quick and dirty deliveries, and unrealistic deadlines. After many long days, nights, weeks, months, and weekends, things actually start to come together. The customer base starts to grow and the orders—and revenue—start rolling in. After a while, the number of employees are in double figures and the founders have become directors.

So, what has this got to do with CD or DevOps? Well, everything really. The culture, default behaviors, and engineering practices of a small software house are what would be classed as pretty good in terms of CD and DevOps. For example:

• There are next to no barriers between developers and the operations teams—in fact, they are generally one and the same
• Developers have full access to the production environment and can closely monitor their software
• All areas of the business are focused on the same thing, that being to get software into the production environment a quickly as possible, thus delighting customers
• Speed of delivery is crucial
• If things break, everyone swarms around to help fix the problem—even out of hours
• The software evolves quickly and features are added in incremental chunks
• The ways of working are normally very agile
• Communication and collaboration across the business is efficient and, for the most part, effective
  

There is a reason for stating that the culture, default behaviors, and engineering practices of a small software house would be classed as pretty good rather than ideal. This is because there are many flaws in the way a small software business typically has to operate to stay alive:

• Corners will be cut to hit deadlines, which compromises software design, quality, and elegance
• Application security best practices are given short shrift or even ignored
• Engineering best practices are compromised to hit deadlines
• The concept of technical debt is pretty much ignored
• Testing won't be in the forefront of the developer's mind (even if it were, there may not be enough time to work in a test-first way)
• Source-and version-control systems are not used religiously
• With unrestricted access to the production environment, ad hoc and uncontrolled tweaks and changes can be made to the infrastructure and environmental setup
• Software releasing will be mainly manual and most of the time an afterthought of the overall system design
• At times, a rough and ready prototype may become production code without the opportunity for refactoring
• Documentation is scant or non-existent—any that does exist is probably out of date
• The work-life balance for an engineer working within a small software house is not sustainable and burn out does happen
  

Let's have a look at the software-delivery process for ACME systems Version 1.0, which, to be honest, shouldn't take too long.

The following diagram gives an overview of the simple process used by ACME systems to deliver software. It's simple, elegant (in a rough-and-ready kind of way), and easy to communicate and understand:

Let's move forward a few years and see how ACME systems is doing, and gain some insight into the benefits and pitfalls of being the leader of the field.

The business has grown in both size and turnover. The customer base is now global and the ACME software platform is being used by millions of customers on a daily basis. ACME systems as a business is well-established, well-renowned, and recognized as being at the forefront in their area of expertise. However, the level of growth and investment has had an impact on profits—which are still pretty much non-existent.

The board of ACME systems are approached by a larger competitor with an acquisition offer. The board and investors feel this makes good commercial sense and that this will help stabilize the business for the future so the sale is agreed. On the whole, everyone is happy with the deal and most see this as positive recognition that they have at last reached the big time.

At first everything is good—everything is great, in fact. The ACME systems team now has the backing it needs to invest in the business and be able to scale out and obtain a truly global reach. It can also focus on the important things, such as building quality software, scaling out the software platform, investing in new technologies, tools, and R&D. The drier side of business—administration, program and project management, sales, marketing, and so on—can be passed to the new parent company that has all of this in place already.

The ACME engineering team moves forward in excited anticipation. The level of investment is such that the software engineering team doubles in size in a number of months. The R&D team—as it's now called—introduces new development tools and processes to enable the speedy delivery of quality software. Agile is adopted across the R&D team, and the opportunity to fully exploit engineering best practices is realized. The original ACME platform starts to creak and is showing its age, so further investment is provided to re-architect and rewrite the software platform using the latest architectural approaches and technologies. In short, the R&D team feels that it's all starting to come together and it has the opportunity to do things right.

In parallel to this, the ACME engineering team members who looked after the production environments are absorbed into the parent's global operations organization. On the face of it, this seems a very good idea; there are datacenters filled with cutting-edge kit, cloud capabilities, global network capabilities, and scalable infrastructure. Everything that is needed to host and run the ACME platform is there. Like the R&D team, the operations team has more than they could have dreamed of. In addition to the tin and string, the operations team also has resources available to help maintain quality, control change to the platform, and ensure the platform is stable and available 24 x 7.

Sitting above all of this, the parent company also has well-established governance, and program—and project-management functions to control and coordinate the overall end-to-end product delivery schedule and process.

On the face of it, everything seems rosy and the teams are working more effectively than ever. At first, this is true, but very soon things start to take a downward turn. Under the surface, things are not all that rosy:

• It is getting increasingly difficult to deliver software—what took days now takes weeks or even months
• Releases are getting overly complex and larger as the new ACME platform rapidly grows and more features are added and changes are made
• Despite the advances in re-architecting the ACME platform, there still remain large sections of buggy legacy code deep within the bowels of the system, which refuses to die
• The R&D team members are now so far removed from the production environment that they are ignorant as to how the software they are writing functions or performs, once it eventually goes live
• The operations team members are now so far removed from the development process that they are ignorant to what's being delivered and how it's being developed
• There are many corporate hoops to jump through and process hurdles to overcome before software changes can go anywhere near the production servers
• Quality is starting to suffer as last-minute changes and frantic bug fixes are being applied to fit into release cycles
• Technical debt amassed during the fast and loose days is starting to cause major issues
• More and more R&D resources are being applied to assist in releases, which is impacting the development of new features
• Deployments are causing prolonged production downtime—both planned and unplanned
• Deadlines are being missed, stakeholders are being let down, and trust is being eroded
• The once-glowing reputation is being tarnished
  

The main problem here, however, is that this attrition has been happening very slowly over a number of months and not everyone has noticed—they're all too busy trying to deliver.

Let's now revisit the process flow for delivering software and see what's changed since last we looked—it's not a pretty picture.

As you can see from the following diagram, things have become very complicated for the ACME team. What was simple and elegant has become complex, convoluted, and highly inefficient. The number of steps and barriers has increased, making it extremely difficult to get software delivered. In fact, it's increasingly difficult to get anything done. The following figure gives you an overview of the ACME Version 2.0 software-delivery process:

If I'm honest, the process as depicted is actually missing some additional detail in relation to the change-management hoops that can add more complexity, effort, and pain. Let's add this detail and look again:

As you can see, things are far from ideal. What was efficient and effective is now the exact opposite. More importantly, the dialogue, quality of the communication, and trust between all of those involved in delivering changes is at best fragmented and pretty much non-existent at worst. What used to be a five-minute chat over a coffee is now a 20-page email thread, meetings, and Skype chats. The ex-ACME engineering team members are less like colleagues and more like entrenched combatants.

Not only is the process long-winded, but the chances of a single change getting all the way through the process without issue is very slim—most of the time, changes have to go around the loop a number of times before they can be classed as shippable; for example, a defect found within any part of the process may push the change all the way back to the beginning of the process.

I mention dialogue, quality of the communication, and trust for a very specific reason—most of the things you read about and hear in relation to CD and DevOps seem to imply that some new tooling and best-of-breed architectural approaches will give you what you need. While this can help, it can also massively hinder—especially when trying to bring these changes on board whilst a business is going through organizational changes and/or growth. In the ACME example, too much was changing too quickly for everyone to understand what was going on and where the journey would end. This inevitably lead to human nature kicking in and people building up barriers and silos to add some stability within the chaos.

If you were to take all of this into account, from an outsider's perspective, the process(es) ACME systems uses to deliver software is now, for all intents and purposes, broken.

OK, so this may be a little over the top, but it just goes to highlight how having a relative chasm between those involved in the delivery of changes—especially the R&D team members (who are tasked with delivering much-needed changes and features) and the operations team members (who are tasked with supporting the live environment into which the changes will be applied)—can completely derail things.

As was previously stated, not everyone noticed the attrition within the organization—luckily, a few observant souls did. A small number of the ACME team's members were aware things are not great and decided to step back and look at things from an outsider's perspective. They then started to see the issues within the overall process as clear as day and became determined to expose these issues for all to see. In addition, they decided to sort the issues out—there was just the small problem of how to do this while everyone was going at full pelt to get software delivered at all costs in their own silos with their own problems.

At first, they invested a vast amount of personal time into investigating and building rough and ready tools, including build and test automation, Continuous Integration (CI), a continuous-deployment pipeline, and system-monitoring solutions. The intention was to automate as many parts of the broken process as possible to reduce the pain. They also applied energies evangelizing within their technically-focused peer groups. Although their ideas and suggestions were welcomed by the majority, there was not the appetite to adopt these new-fangled tools—everyone was far too busy trying to ship software within the broken process. They needed another way.

They decided that they needed some assistance, so they sought out a like-minded manager with influence within the wider business who could help them get some much-needed traction. After much cajoling, discussions, and pleading, the manager agreed to help them to obtain budget to form a small team focusing on advancing the CD and DevOps tooling. The newly-formed team's members spent a few months identifying and breaking down the immediate and most painful issues, and built, installed, and implemented tooling to remove some of the pain—to ease the adoption, many of the tools are bespoke to fit into the existing processes.

This went some way to address the broken process but the reality is that the tools did not have the impact they envisaged. In fact, the tools themselves needed to be altered so much to fit the existing process that they started to become unreliable and too complex, so much so that those who were originally behind the approach started to question the validity of their decisions.

Ultimately, there is a much bigger issue that tooling cannot address—the culture of the organization itself, the behaviors of those within it, and the many disjointed methods of communication between the disconnected silos that had formed over the years. It became obvious that all the tools and tea in China will not bring pain relief; something more drastic was needed.

The team's members refocused and soon realized that it's not the tools that need to change to fit the process, but the process and ways of working that needs to change. If this was addressed, the tools could simply be taken off the shelf—so to speak—and used without extensive modification. The team's members have to drastically change their direction, become less technology-focused, and act more like agents for business change. They then highlighted this now-obvious fact to as many people as they can up and down the organization while the influential manager worked to obtain backing from the senior leadership to implement far-reaching business change. Luckily, their reputation and standing within the organization was such that getting backing was easy.

We're now going on to the third stage of the evolution, where things start to come back together and the ACME team regains their ability to deliver quality software when it is needed.

Now that the CD and DevOps team has official backing from high up, its members start to address the broken culture and behaviors, and develop ways to overcome and/or remove the barriers. They are no longer simply a technical team; they are a catalyst for business change.

The remit is clear—do whatever is needed to streamline the process of software delivery and make it seamless and repeatable. In essence, implement what we now commonly refer to as CD and DevOps.

The first thing they do is to go out and talk with as many people across the business as possible to ensure they are also aware of the broken process and its root causes. Simply put, if someone is actively involved in the decision-making process of getting software from conception to consumer, or involved in supporting it when it's live, they are a chat target. This not only gathers useful information, but also gives the team the opportunity to evangelize and form a wider network of like-minded individuals.

The team has a vision, a purpose, that its members passionately believe in what needs to be done, and they have the energy and drive to do it.

Over the next few months, they embark on (among other things):

• Running various in-depth sessions to understand and map out the end-to-end product-delivery process
• Refining and simplifying the tooling based upon continuous feedback from those using it—where applicable, replacing in-house built solutions with off-the-shelf ones
• Addressing the complexity of managing dependencies and the order of deployment
• Engaging experts in the field of CD and DevOps to independently assess the progress being made (or not, as the case may be)
• Arranging offsite CD and DevOps training and encouraging both R&D and Ops team members to attend the training together (it's amazing how much DevOps collaboration stems from a chat in the hotel bar)
• Reducing the many handover and decision-making points throughout the software-release process
• Removing the barriers to allow developers to safely deploy their own software to the production platform
• Working with other business functions to gain trust and help them to refine and streamline their processes
• Removing the us-and-them attitudes and behaviors, and reinforcing trust-based relationships
  
• Working with R&D and operations teams to experiment with different agile methodologies, such as Kanban, scrum, and lean
• Openly and transparently sharing information and data around deliveries and progress being made across all areas of the business
• Replacing the need for complex performance-testing with the ability for developers to closely monitor their own software running in the production environment
• Removing the need for downtime to release changes
• Evangelizing across all areas of the business to share and sell the overall vision and value of CD and DevOps

This is by no means a walk in the park and it takes determination, steadfast focus, patience, and, above all, time to produce quantifiable, positive results, however after some months, the vision and benefits start to be realized. Now the process of building and delivering software has transformed to the extent that a code change can be built, fully tested, and deployed to the production platform in minutes many times per day—all at the press of a button and initiated and monitored by the developer who made the change, all with no downtime and little/no impact on the customers. The stakeholders have a trusted and reliable way of delivering value to their customers, the R&D team has the tooling and empowerment to deliver value as and when it is needed, and the Ops team has a stable and reliable platform that it can support and optimize.

Let's look again at the software-delivery process flow to see what results have been realized.

As you can see from the diagram, the process looks much healthier. It's not as simple as Version 1.0, but it is efficient, reliable, and repeatable. Some much-needed checks and balances have been retained from Version 2.0 and optimized to enhance rather than impede the overall process:

This highly efficient process has freed up valuable R&D and operations resources so that they can focus on what they are best at—developing and delivering new high-quality features, and ensuring that the production platform is healthy and customers are delighted.

The ACME systems team has got back its mojo and is moving forward with a newfound confidence and drive. It now has the best of both worlds, and there's nothing stopping it.

The ACME systems team's members have come through their challenges stronger and leaner but their story doesn't end there. As with any successful business, they don't rest on their laurels but decide to expand into new markets and opportunities—namely, to build and deliver mobile-optimized clients to work with and complement their core web-based propositions.

With all they have learned throughout their evolution, they know they have an optimal way of working to allow them to deliver quality products that customers want, when they want them, and they know how to deliver quickly, reliably, and incrementally. However, the complexities of delivering features to a hosted web-based platform are not the same as the complexities of delivering features to an end consumer's mobile device—they are comparable but not the same. For example, the process of delivering code to production servers many times per day is under the control of the ACME team, whereas they have little or no control over how their mobile clients are delivered to end customers, nor if and when the end customer will install the latest and greatest version from the various app stores onto which the mobile client is published. In addition to this, the production platform onto which the mobile client will be installed is pretty much an unknown in terms of spec, performance, and storage.

All is not lost, though—far from it. The members of the ACME systems team have learned a vast amount throughout their evolutionary journey, and decide to approach this new challenge as they had done previously. They know they can build, test, and deliver software with consistent quality. They know how to deliver change incrementally with little or no impact. They know how to support customers and monitor and react quickly to change. They know their culture is mature and that the wider organization can work as one to overcome shared challenges.

As the new venture progresses, they also discover another side-effect of their newly rekindled success: the amount of traffic and transactions start to grow very quickly. They therefore need to scale out their platform and they need to do it as soon as possible. Rather than rely on their own datacenters, they decide to move their entire platform to a globally-distributed cloud-based solution. This brings with it new challenges: the infrastructure is completely different, the provisioning tools are new, the tools used to build and deliver software are incompatible with the existing ACME tools. Again, the ACME systems team take this in stride and forge ahead with confidence using the highly collaborative ways of working, techniques, and approaches that are now part of their DNA.

Let's look at what this all means from a holistic point of view.

Throughout this chapter, we have been following the evolution of ACME systems: where they started, the growing pains that came from success, how they discovered that rapid growth brings with it negatives as well as positives, how they overcame their near extinction by adopting CD and DevOps, and how they regained their mojo and confidence to move forward. All of this can be represented by the following simple diagram:

What they also learned—somewhat late in the evolution—was that CD, and DevOps-adoption has little to do with technical tools and everything to do with how people work together. Without the changes to the culture and behaviors of everyone involved in the end-to-end delivery process, it is almost impossible to realize and maximize the benefits that a successful adoption of CD and DevOps brings. It could be said that if they knew this simple, yet mostly overlooked, fact from day one, then the adoption would have happened sooner and the business would have been far stronger far sooner. Hopefully, this will provide some food for thought for you as you move through the rest of the book.

At the beginning of this chapter, I stated that the effectiveness of adopting CD and DevOps is very much dependent on where a business sits on the evolutionary scale. We've been through ACME's evolution and the phases it went through. Please take into account that ACME is fictional and its story is pretty simplistic. A real-world business is not simple—far from it—and it is pretty difficult to ascertain where a given business sits on the CD and DevOps evolutionary scale. Without this information, it's hard to understand how receptive, responsive, and open to adoption a business actually is.

With that said, there are some simple ways of getting a clearer idea. For example, the following list of questions can help you get a rough idea. Looking at your business, ask yourself the following:

The preceding is simply an example that gives you and insight into how you can—at a very holistic viewpoint—ascertain how mature the business is and where it sits within the CD and DevOps evolutionary scale. You will no doubt have some additional, complimentary, or more relevant questions you could use. However, if you follow a similar format, you will be able to get a feel for where things sit, and more importantly, what areas need the most focus. You should widen the net as much as possible to get a view from as many parts of your business as possible—that way, you won't come across surprises when you decide to take the plunge.

The ACME systems evolution story is not atypical of the many software businesses out there today. No doubt, you will recognize and relate to some of the traits and challenges detailed in the ACME journey, and you should now be able to plot where your business (or your client's/partner's business) currently sits within the CD and DevOps evolutionary scale. You also got a holistic view of what CD and DevOps is and what it isn't.

We'll now move from storytelling mode and start to look in more detail at some of the practical aspects of adopting CD and DevOps, starting with how you identify the underlying problems that can (and do) stifle the delivery of quality software.

In Chapter 2, Understanding Your Current Pain Points, we'll be looking into how you go about identifying the problems and issues within their Software Delivery Life Cycle (SDLC) and highlight some tools, techniques, and approaches to surface said problems and issues so that they can be fixed.