ACME systems evolution phase 2.0

The business has grown in both size and turnover. The customer base is now global and the ACME software platform is being used by millions of customers on a daily basis. ACME systems as a business is well-established, well-renowned, and recognized as being at the forefront in their area of expertise. However, the level of growth and investment has had an impact on profits—which are still pretty much non-existent.

The board of ACME systems are approached by a larger competitor with an acquisition offer. The board and investors feel this makes good commercial sense and that this will help stabilize the business for the future so the sale is agreed. On the whole, everyone is happy with the deal and most see this as positive recognition that they have at last reached the big time.

At first everything is good—everything is great, in fact. The ACME systems team now has the backing it needs to invest in the business and be able to scale out and obtain a truly global reach. It can also focus on the important things, such as building quality software, scaling out the software platform, investing in new technologies, tools, and R&D. The drier side of business—administration, program and project management, sales, marketing, and so on—can be passed to the new parent company that has all of this in place already.

The ACME engineering team moves forward in excited anticipation. The level of investment is such that the software engineering team doubles in size in a number of months. The R&D team—as it's now called—introduces new development tools and processes to enable the speedy delivery of quality software. Agile is adopted across the R&D team, and the opportunity to fully exploit engineering best practices is realized. The original ACME platform starts to creak and is showing its age, so further investment is provided to re-architect and rewrite the software platform using the latest architectural approaches and technologies. In short, the R&D team feels that it's all starting to come together and it has the opportunity to do things right.

In parallel to this, the ACME engineering team members who looked after the production environments are absorbed into the parent's global operations organization. On the face of it, this seems a very good idea; there are datacenters filled with cutting-edge kit, cloud capabilities, global network capabilities, and scalable infrastructure. Everything that is needed to host and run the ACME platform is there. Like the R&D team, the operations team has more than they could have dreamed of. In addition to the tin and string, the operations team also has resources available to help maintain quality, control change to the platform, and ensure the platform is stable and available 24 x 7.

Sitting above all of this, the parent company also has well-established governance, and program—and project-management functions to control and coordinate the overall end-to-end product delivery schedule and process.

On the face of it, everything seems rosy and the teams are working more effectively than ever. At first, this is true, but very soon things start to take a downward turn. Under the surface, things are not all that rosy:

• It is getting increasingly difficult to deliver software—what took days now takes weeks or even months
• Releases are getting overly complex and larger as the new ACME platform rapidly grows and more features are added and changes are made
• Despite the advances in re-architecting the ACME platform, there still remain large sections of buggy legacy code deep within the bowels of the system, which refuses to die
• The R&D team members are now so far removed from the production environment that they are ignorant as to how the software they are writing functions or performs, once it eventually goes live
• The operations team members are now so far removed from the development process that they are ignorant to what's being delivered and how it's being developed
• There are many corporate hoops to jump through and process hurdles to overcome before software changes can go anywhere near the production servers
• Quality is starting to suffer as last-minute changes and frantic bug fixes are being applied to fit into release cycles
• Technical debt amassed during the fast and loose days is starting to cause major issues
• More and more R&D resources are being applied to assist in releases, which is impacting the development of new features
• Deployments are causing prolonged production downtime—both planned and unplanned
• Deadlines are being missed, stakeholders are being let down, and trust is being eroded
• The once-glowing reputation is being tarnished
  

The main problem here, however, is that this attrition has been happening very slowly over a number of months and not everyone has noticed—they're all too busy trying to deliver.

Let's now revisit the process flow for delivering software and see what's changed since last we looked—it's not a pretty picture.

Software-delivery process flow Version 2.0

As you can see from the following diagram, things have become very complicated for the ACME team. What was simple and elegant has become complex, convoluted, and highly inefficient. The number of steps and barriers has increased, making it extremely difficult to get software delivered. In fact, it's increasingly difficult to get anything done. The following figure gives you an overview of the ACME Version 2.0 software-delivery process:

Overview of ACME Version 2.0 software-delivery process

This far-from-simple process is something that large software businesses will recognize. Looking again from a CD and DevOps perspective, this process is far from ideal as there are now many barriers between those delivering software and those supporting it.

If I'm honest, the process as depicted is actually missing some additional detail in relation to the change-management hoops that can add more complexity, effort, and pain. Let's add this detail and look again:

More realistic overview of ACME Version 2.0 software-delivery process

As you can see, things are far from ideal. What was efficient and effective is now the exact opposite. More importantly, the dialogue, quality of the communication, and trust between all of those involved in delivering changes is at best fragmented and pretty much non-existent at worst. What used to be a five-minute chat over a coffee is now a 20-page email thread, meetings, and Skype chats. The ex-ACME engineering team members are less like colleagues and more like entrenched combatants.

Not only is the process long-winded, but the chances of a single change getting all the way through the process without issue is very slim—most of the time, changes have to go around the loop a number of times before they can be classed as shippable; for example, a defect found within any part of the process may push the change all the way back to the beginning of the process.

I mention dialogue, quality of the communication, and trust for a very specific reason—most of the things you read about and hear in relation to CD and DevOps seem to imply that some new tooling and best-of-breed architectural approaches will give you what you need. While this can help, it can also massively hinder—especially when trying to bring these changes on board whilst a business is going through organizational changes and/or growth. In the ACME example, too much was changing too quickly for everyone to understand what was going on and where the journey would end. This inevitably lead to human nature kicking in and people building up barriers and silos to add some stability within the chaos.

If you were to take all of this into account, from an outsider's perspective, the process(es) ACME systems uses to deliver software is now, for all intents and purposes, broken.

OK, so this may be a little over the top, but it just goes to highlight how having a relative chasm between those involved in the delivery of changes—especially the R&D team members (who are tasked with delivering much-needed changes and features) and the operations team members (who are tasked with supporting the live environment into which the changes will be applied)—can completely derail things.

An outsider's perspective from the inside

As was previously stated, not everyone noticed the attrition within the organization—luckily, a few observant souls did. A small number of the ACME team's members were aware things are not great and decided to step back and look at things from an outsider's perspective. They then started to see the issues within the overall process as clear as day and became determined to expose these issues for all to see. In addition, they decided to sort the issues out—there was just the small problem of how to do this while everyone was going at full pelt to get software delivered at all costs in their own silos with their own problems.

At first, they invested a vast amount of personal time into investigating and building rough and ready tools, including build and test automation, Continuous Integration (CI), a continuous-deployment pipeline, and system-monitoring solutions. The intention was to automate as many parts of the broken process as possible to reduce the pain. They also applied energies evangelizing within their technically-focused peer groups. Although their ideas and suggestions were welcomed by the majority, there was not the appetite to adopt these new-fangled tools—everyone was far too busy trying to ship software within the broken process. They needed another way.

They decided that they needed some assistance, so they sought out a like-minded manager with influence within the wider business who could help them get some much-needed traction. After much cajoling, discussions, and pleading, the manager agreed to help them to obtain budget to form a small team focusing on advancing the CD and DevOps tooling. The newly-formed team's members spent a few months identifying and breaking down the immediate and most painful issues, and built, installed, and implemented tooling to remove some of the pain—to ease the adoption, many of the tools are bespoke to fit into the existing processes.

This went some way to address the broken process but the reality is that the tools did not have the impact they envisaged. In fact, the tools themselves needed to be altered so much to fit the existing process that they started to become unreliable and too complex, so much so that those who were originally behind the approach started to question the validity of their decisions.

Ultimately, there is a much bigger issue that tooling cannot address—the culture of the organization itself, the behaviors of those within it, and the many disjointed methods of communication between the disconnected silos that had formed over the years. It became obvious that all the tools and tea in China will not bring pain relief; something more drastic was needed.

The team's members refocused and soon realized that it's not the tools that need to change to fit the process, but the process and ways of working that needs to change. If this was addressed, the tools could simply be taken off the shelf—so to speak—and used without extensive modification. The team's members have to drastically change their direction, become less technology-focused, and act more like agents for business change. They then highlighted this now-obvious fact to as many people as they can up and down the organization while the influential manager worked to obtain backing from the senior leadership to implement far-reaching business change. Luckily, their reputation and standing within the organization was such that getting backing was easy.

We're now going on to the third stage of the evolution, where things start to come back together and the ACME team regains their ability to deliver quality software when it is needed.