Most developers and architects will be familiar with the concept of a pattern – a good solution to a recurring problem within an architectural domain described in a formalized and reusable way. Some classic examples include the following:

• Singleton: A software design pattern that limits the number of instances of a given type to one.
• Fire-and-forget: An asynchronous integration pattern that sends off a message from a computational context and proceeds without waiting for a response.
• Model-View-Controller (MVC): An architectural pattern that divides an application into three tiers with specifically defined responsibilities:
  1. First, a model maintains the state of the application and is responsible for any changes to data
  2. Second, a view shows a particular representation of that model to an end user via some interface
  3. Third, a controller implements the business logic that responds to events in the user interface or changes in the model and does the necessary mediation between the view and the model

This pattern is shown in the following diagram:

Figure 1.1 – MVC pattern diagram

Patterns such as these have been defined at many levels of abstraction and for many different platforms.

References

You can look at the following resources to get a good introduction to the various patterns that one can apply from a Salesforce perspective.

The integration patterns guide lists all the main patterns to use when designing Salesforce integrations: https://developer.salesforce.com/docs/atlas.en-us.integration_patterns_and_practices.meta/integration_patterns_and_practices/integ_pat_intro_overview.htm. In a Salesforce world, this may be the most commonly referenced set of patterns as they are ubiquitous for integration design.

The book Apex Design Patterns from Packt, by Anshul Verma and Jitendra Zaa, provides patterns at the software design level and the concrete code level for the Apex language.

The Salesforce Architects site, while new, contains a range of patterns across domains, from code-level specifics to reference architectures and solution kits to good patterns for selecting governance: https://architect.salesforce.com/design/#design-patterns.

The point is that we have lots of good patterns to choose from on the Salesforce platform, many that are provided by Salesforce themselves, others by the wider community. Many patterns that apply to other platforms are also relevant to us and we learn much by studying them.

But this is a book about anti-patterns, not patterns. So why am I starting with a discussion about patterns? It turns out that the two are nearly inseparable and originate in the same tradition. Understanding anti-patterns, therefore, begins with an understanding of what a pattern is. Indeed, one common form of anti-patterns is a design pattern that has been misapplied. We will explore this in the next section.

From pattern to anti-pattern

The design pattern movement in software architecture originates from the work of Christopher Alexander, an architect whose job was to design buildings rather than systems. In his work The Timeless Way of Building, he introduced a pattern template for physical architecture that consisted of a name, a problem statement, and a solution in a standardized format.

The rhetorical structure provided by the Alexandrian template was rapidly adopted by architects of a different kind, the ones that build software. They recognized the power of a standardized way to describe problem-solution sets for communicating good practice. With the publication of the classic Gang of Four book, Design Patterns: Elements of Reusable Object-Oriented Software, the use of patterns became mainstream within software development and remains so to this day.

The research on patterns inspired an incipient community of practitioners and researchers in software engineering to think about failure modes of software systems analogously to how design patterns were being used. This happened over an extended period of time, and it isn’t possible to point to anyone in the anti-patterns movement that can be seen as the genuinely foundational figure.

However, many research papers on the topic start with the definition given by Andrew Koening in the Journal of Object-Oriented Programming in 1995. This definition says that an anti-pattern is very similar to a pattern and can be confused for one. However, using it does not lead to a solution, but instead has negative consequences.

That definition captures much of the essence and can be usefully combined with the following thoughts from Jim Coplien, another early pioneer. He thought that good patterns in and of themselves were not sufficient to define a successful system. You also have to be able to show that anti-patterns are absent.

In a nutshell, then, an anti-pattern is a pattern that occurs in unsuccessful software systems or projects that can look superficially like a good solution but in practice gets you into trouble. Some common anti-patterns that have been around for ages and are still relevant include the following:

• Stovepipe: A system or module that is effectively impossible to change because of how its interfaces are designed. See the following diagram for an illustration:

Figure 1.2 – The Stovepipe anti-pattern

• Blob: A design where a single class effectively encapsulates all the business logic and functionality, leading to major maintenance headaches.
• Intensive coupling: A design that uses an excessively large number of methods from another class or module in implementing a feature, leading to a deep dependency that is hard to understand or change.

We’ll dig into all these anti-patterns from a Salesforce perspective in later chapters, underscoring the unfortunate fact that Salesforce, while a great platform, is still a software system prone to the kind of mistakes that have plagued software systems for decades. If this were not the case, then, of course, there’d be no need for this book or, for that matter, for Salesforce architects.

Having discussed the historical origins of anti-patterns, we will now discuss how they arise in the real world.

Where do anti-patterns come from?

Anti-patterns tend to arise again and again because the circumstances that drive technology projects into trouble are at least somewhat predictable. Equally predictable are the responses of people put into those situations and as people in tough spots frequently make the same bad decisions, we get systematic patterns to study.

Generally speaking, the most important thing to remember about anti-patterns is that they seem like a good idea at the time. They aren’t simply bad practices that someone should know better than to apply. You can make a reasoned argument that they should lead to good or at least acceptable outcomes when applied.

Sometimes the decision to apply an anti-pattern is down to inexperience, and sometimes it is down to desperation. But as often as not, it is down to experienced professionals convincing themselves that in this case, doing what they’re doing is actually the right call or that this situation really is different.

We will try to reflect this diversity of origins in the examples we show throughout this book. However, to do this, we first need to show how we will present the examples in a consistent way to enable learning.

An anti-pattern template

One of the key characteristics of both patterns and anti-patterns is that they are written using a recognizable template. Many templates have been suggested over the course of the years, some more elaborate than others.

The template we will use in this book contains the bare bones that recur in nearly all existing anti-pattern templates. However, we do not include a great number of optional elements. Most additional elements that are included in other templates serve to ease categorization or cross-referenceability. Those are highly desirable elements to have when creating a searchable catalog of patterns or anti-patterns but are less useful in a printed book. Therefore, we omit them and instead include the following elements:

• Name: An easy-to-remember moniker for the pattern that serves to identify it uniquely and help discourse among architects.
• Type: In this book, we categorize our anti-patterns by the domain from the Certified Technical Architect (CTA) review board examination that they are relevant to. This is both to help people on the CTA journey, but also because this is a familiar typology for Salesforce architects.
• Example: We will introduce each anti-pattern by giving an example of how it might occur. The examples will be fictional but grounded in real-world events. This will frame the anti-pattern and give you an immediate understanding of the issues involved before delving deep into the nuts and bolts.
• Problem: This section describes more formally the problem or problems that the anti-pattern purports to solve. These are the real issues that the anti-pattern is meant to be a solution for, although using it in practice will turn out to be a bad idea.
• Proposed solution: How the anti-pattern claims to solve the problem described in the previous section and how that can be tempting to believe given certain circumstances.
• Results: In the results section, we outline what results you can expect from applying the anti-pattern along with its main variations. We go into the detail of why this is a bad solution, although it might look good at the outset.
• Better solutions: The final section in the template will tell you what to do instead when faced with the problem that is the basis for the anti-pattern. Not all problems have easy solutions, but you can generally do better than applying an anti-pattern.

Now that we have an understanding of what anti-patterns are and how they are going to be structured in this book, we will move on to explaining how you, as an architect, can improve your skills by using them to learn.

While a theoretical understanding of what patterns and anti-patterns are and how they might generally be helpful is great, you are no doubt looking for more from this book than just a theoretical discussion.

In this section, we will go through the value of learning from mistakes, both your own and other people’s, and show you how we are going to use anti-patterns to sharpen your architecture chops in the context of a concrete example.

How great architects learn from mistakes

We work in an industry where failure is a normal occurrence. The Standish Group CHAOS report, which is the most commonly cited source on these matters, in 2020 estimated that 66% of all technology projects globally ended in partial or complete failure.

This is a little bit better than when I started my career more than 20 years ago. Then, the figure stood in the low 70s. However, while things have improved, they have only improved a little bit, despite agile development, cloud computing, artificial intelligence, and great software platforms such as Salesforce.

This is disheartening, but only proves the point of one of my personal heroes, Fred Brooks, that “The complexity of software is an essential property, not an accidental one," which unfortunately means that we will never find a silver bullet to solve all problems in software architecture and design. Instead, we are faced with the hard work of learning how to manage this complexity in a reasonable way. To do so is the principal job of an architect.

It stands to reason that in an area with high failure rates and irreducible complexity, we need to have good rules and guidelines to keep us on the right path. That is what patterns and best practices are for. They are indispensable, but they are not enough.

To become great at our profession and to be able to design software that bucks the industry trend, we need not just learn from our own failures but also from the vast repository of failed projects and bad practices we see all around us.

That usually isn’t hard to do. Many times, when the architect is brought into a project, it is because there is already an issue to fix. Using these occurrences as learning opportunities and analyzing them with that view in mind can be greatly rewarding.

However, there is a step further to go in this direction, which is what anti-patterns offer. They encompass the ways in which things frequently go wrong in a way that allows both post-hoc learning and also real-time intervention.

The thing is that while projects go wrong, they don’t do so randomly. There are systematic patterns that repeat time and again. Learning how things go wrong in systematic ways can give you an entirely new set of responses in your toolbox that you can deploy to help make your project one of the 34% that don’t fail.

We will start that learning journey with an explanatory example.

An example: The Hero anti-pattern

There is no better way to start learning than using an example. We won’t have the chance to cover many general management-level anti-patterns in this book, so I will use one of the classics in this genre to show you how the template works and how to read it to get the most out of it.

First, we will present the anti-pattern and then provide an analysis of what we can learn from it.

Hero (development life cycle and deployment planning)

Tom is the project manager for a large greenfield Salesforce implementation in the manufacturing industry working with Sales and Service Cloud for 2,000 users. The project is meant to be quite simple, a basic MVP to get the platform off the ground, and that is how it has been scoped and staffed. The project is meant to go into full production after six months with four months of implementation followed by test, training, rollout, and hypercare.

The first three months of implementation fly by and everything on the project management dashboard stays green. The team makes steady progress and relations with the external consultancy that is helping provide specialist resources remain good.

However, when Tom delivers the first round of business reviews, things start to change quickly. It turns out that a lot of detailed requirements have been missed from the initial scope and that many edge cases aren’t covered by the current configuration. The feedback from the business is that they won’t be able to go live with the system unless a large list of additional functionality is included.

Tom goes to the steering committee to ask for more budget and a scheduled increase to accommodate the business demands. The steering committee grants the request for an increased budget but tells him that the schedule is immovable. He must find a way to get it done within the current timeline.

Tom replans the project from the ground up. He can just make it all fit by compressing the testing and cutover plan if he adds some new resources from the external partner and asks for some overtime from his existing team. He sends out the new plan to the team along with a rousing email calling on everyone to rise to the challenge.

Over the course of the next month, the project slips again, and Tom’s new plan is looking less and less likely to succeed. It’s not that anything big goes wrong, but lots of little things just aren’t completed on time or require rework because of misunderstandings. In particular, the new consultants he has brought in from the external partner seem to make a lot of basic mistakes.

Tom calls his boss, the Senior Director for IT, to tell him about the situation and ask for help in getting an extension to the schedule. She tells him that the schedule has been committed to the board of directors of the company and that heads will roll if it is not met. This is the time for the team to pull out all the stops and get it done, she says.

Tom goes back to his team to relay the news and once again calls for everyone to give it everything to get things over the line. Unfortunately, most people are already working as hard as their situations allow. In addition, relations with the external partner have soured and they are not willing to put in additional hours without additional funding, which Tom does not have in the budget.

There are some bright spots, however. In particular, two young members of the technical staff, Kayleigh and Negash, prove willing to go above and beyond in order to get things done. Over the final month of delivery, they work 24/7 on the project with Tom cheering them on.

Figure 1.3 – The dangerous feeling one might have when engaging in the Hero anti-pattern

Between the two of them, they manage to clear away enough of the new features and change requests during the final stretch that Tom feels growing confidence that he will be able to meet enough of the requests for the project launch to not be a disaster. There will be exceptions, but he can find a way of managing those later. As long as the impending go-live goes well, the project can still succeed.

However, User Acceptance Testing (UAT) throws a spanner in the works as major quality issues are discovered. The steering committee holds a crisis meeting that ends up concluding that the go-live will have to be postponed for a week. The team will have to work flat out during this period to fix the issues.

While everyone pitches in, the responsibility falls disproportionately on Kayleigh and Negash, who are both starting to show the strain of the continuous effort. Tom gives them encouragement at every chance and singles them out for public praise. He also promises them a cash bonus and extra holidays when the project is done.

The day for retesting arrives and while many issues have been fixed satisfactorily, there are quite a few remaining issues, including a good number that had previously been fixed and are now recurring.

The steering committee holds another crisis meeting and they take the decision to go ahead with the launch despite the issues. These issues will need to be fixed during the hypercare period, but they can be tolerated for a short amount of time.

The next few weeks of Tom’s, Kayleigh’s, and Negash’s lives happen in a blur of constant motion. They are pulled from escalation to escalation as issues occur, are fixed, and reoccur. Kayleigh and Negash start buckling under the pressure, but with no alternative resources knowing the configuration, they are effectively forced to carry on.

Eventually, the issues settle down. The important bugs are fixed, the business puts in place manual workarounds for the things that were missed, life starts to get back to normal. Tom calls the team for a victory celebration, but it is a muted affair.

After taking their extra holidays, Kayleigh and Negash both accept offers from big consulting companies, leaving the company with no one to support large chunks of functionalities on their newly implemented platform.

Problem

The Hero anti-pattern generally purports to fix an urgent delivery problem that has occurred either in a project context, as in our example, or during normal operations. When it occurs in normal operational mode, this is often in a context where firefighting issues with the system have become a run-of-the-mill occurrence.

Usually, the problem occurs in a context characterized by some of the following factors:

• There are limited resources to carry out the work needed to fix the urgent problem and there are good reasons why new resources cannot be brought in at this time.
• The project has a tight schedule that is perceived to be set in stone or the issue is live, critical, and affecting important business users adversely in ways that cause a lot of noise.
• There is knowledge about the problem concentrated in a small number of heads, that is to say, a few people, such as Kayleigh and Negash, who volunteered to take on the role, or frequently a lead developer who is the only one with the technical knowledge to fix the issue at the given time.
• The situation is considered somehow special: either this is a new project and there isn’t a precedent, or the issue is considered so unique that you can’t really plan for it.
• The crisis element is often quite visible in situations that foster the Hero anti-pattern. Sometimes, important parts of the company’s future success or even survival are brought into play.

These factors can all make the problem seem more important to fix in a limited time scale and make the Hero option seem attractive.

Proposed solution

The Hero anti-pattern proposes to solve the problem described in the preceding section by allowing an individual or a small group to take too much responsibility for resolving it effectively by working as much as is required, even at some cost to themselves, to get things done.

This can be attractive both to management and to the people involved for a variety of reasons:

• The effort does tend to produce some results in the short term, giving a sense of momentum and success.
• Everyone, or at least nearly everyone, wants to be a hero and be singled out for praise and rewards. To some people, that is worth the inconvenience of the additional effort.
• It is always possible to imagine that the current situation is somehow unique and not reflective of a deeper problem with process or culture within the organization, thereby justifying what is done as exceptional.
• Even if we acknowledge that there are underlying issues, often these can be put out of mind as something to be dealt with later. Of course, in organizations that rely on the Hero anti-pattern, later never comes.

There are several common variants of the Hero anti-pattern that are worth mentioning:

• Superman, a variant where someone, usually a senior technical person, is glorified and held up as the only person who can fix serious issues with a given system. Often, this myth becomes self-perpetuating.
• Rookies, the variant seen in the example, where junior team members take on extra responsibilities in an effort to step up to the challenge that is being presented to them.
• No Time for Knowledge Transfer, a situation where heroics are required by a seemingly never-ending time crunch that would make it possible for the hero or heroes to transfer required knowledge to others.

While this anti-pattern is clearly seductive, and many of us have fallen prey to it several times over the course of our careers, it almost invariably has negative long-term consequences, which we’ll explore next.

Results

While the Hero anti-pattern tends to give good short-term results, which is a major source of its enduring appeal, there is a long list of negative results that tend to accumulate over time in organizations that rely on this anti-pattern to get things done.

Some of the most common negative results include the following:

• The creation of a single point of failure that increases risks to an organization substantially, should the Hero fall under the proverbial bus, and gives the Hero a lot of leverage in negotiations with the organization.
• The Hero, over time, will start to feel the pressure, as Kayleigh and Negash did in our example, but will have very limited options to change the situation. This situation is highly conducive to burnout, which brings with it all the problems of the first point as well as the risk of the Hero making serious errors due to the strain.
• Heroes don’t scale. That is to say, the organization won’t be able to deploy projects at a bandwidth that is wider than what the Hero can accommodate. This can be seriously limiting to new initiatives in some cases.
• Heroes aren’t replicable. You can’t easily replicate the Hero or their special powers and therefore you have limited options for creating a predictable and repeatable process.
• Heroes can accumulate serious technical debt, which may often go unmanaged, because they must do things quickly, under pressure, and without real supervision. This can lead to major maintenance issues in the long term.
• There is low transparency into the process by which Heroes get things done, leading to a lack of predictability and manageability.
• Heroes don’t have time to explain how things were implemented, so there is often poor or entirely missing documentation.
• The rest of the team may feel disempowered, overlooked, and demotivated as all the attention goes to the Heroes, with little opportunity for others to make contributions in a non-heroic way.

You don’t necessarily see all these negative outcomes in all instances of this anti-pattern, and this list unfortunately isn’t exhaustive either. But hopefully, this is enough to make you think twice about applying this anti-pattern and look at better options, which we’ll explore next.

Better solutions

The fundamental problem with the Hero anti-pattern is that you are relying on specific individuals, with often hidden knowledge, working hard – usually too hard – to get things done rather than on repeatable, transparent, and manageable processes that will allow you to continue to deliver, even as the context and the people involved change.

The primary way to get away from the Hero anti-pattern is therefore to work on your processes and spread skills and knowledge systematically across the team. In our example, there were potential issues with scope management, with the initial discovery work, with governance and its understanding of the real issues on the ground, and with the way the project had been structured to go live with a big bang rather than in small increments.

What specific interventions will provide the most leverage will vary a lot between organizations, but some good places to look include the following:

• Moving towards a DevOps culture with smaller incremental releases that have lower risk profiles
• Having multi-functional teams with frequent direct collaboration and peer review to spread knowledge around
• Encouraging and rewarding leads and specialists more for mentoring and bringing up juniors rather than for putting out the latest fire
• Incorporating better risk management and governance in projects to have the right contingencies in place when things go wrong, as they inevitably will
• Challenging the cultural norms that put primacy on delivering big dramatic wins against the odds, rather than on making steady, undramatic, but repeatable progress on a regular basis
• Emphasize roles and processes, not individuals when planning, building, and operating systems, especially when communicating with the wider stakeholder community
• Make the costs of the Hero anti-pattern visible by capturing the technical debt, the risk, and the missed opportunity to be able to replicate efforts that the organization loses by relying on this pattern
• Ensure that detailed requirements and edge cases are planned for when beginning the project, which reduces the probability that you will need a hero

In truth, it is not always possible to completely avoid the Hero anti-pattern. Sometimes, things are on fire and there is only one person around who can fix it. What we need to recognize is that this is not a good situation, but an anti-pattern that we need to work hard to fix so that it doesn’t recur. The more you do this, the less you’ll have to rely on heroes and the fewer fires you’ll have to put out.

Having covered our first example of a real anti-pattern, we will go on to analyze it a little more deeply to see how we can maximize our learning from it.

Analyzing the example

The Hero anti-pattern is a classic and most seasoned IT professionals will have encountered it during their careers. However, interesting as it is, in this book, we are also looking to pull out the larger patterns, we can learn from our examples to hone our architecture skills.

Throughout this book, we will do this by having a section towards the end of a chapter that extracts key learning points for you to take on your future architecture journey. We do this both with a mind to real-life practice, but we also list learning points specifically for those of you who are on the track towards the CTA exam.

Considering the Hero anti-pattern, a few learning points you might extract for real-life practice are as follows:

• When you are faced with a crisis that calls for extraordinary effort on the part of some or all of the team, take the time to step back and consider the process failures that led to this situation. Capture this for future use.
• Relying on a small number of extremely skilled individuals can be dangerous in the long run, even if it’s useful right now.
• The pressure you might feel towards going above and beyond may reflect a culture that doesn’t have its priorities right from a technical point of view. You may want to challenge that if possible.
• Go out of your way to empower and bring up junior staff to avoid being in the position where you have to be the hero.
• Be diligent about advocating for good governance both at the project and technical levels as well as capturing and remedying the technical debt that accumulates from “special” situations.

Looking at the CTA review board, you can note the following lessons:

• Be careful about suggesting big bang delivery approaches. They can be the right choice, but frequently they can lead to the issues highlighted in the example. Prefer using agile, incremental approaches unless there is a specific reason not to.
• Ensure that you do include the necessary governance functions, including Project Management Office (PMO), steering committee, design authority, and maybe a change advisory board. In this example, much could have been avoided if the latter two had been in place.
• Be explicit about risks and risk management. Include risks upfront and be ready to talk about how to manage and mitigate them.

While we will be able to directly pull out many learning points, we also encourage you to go further with this method and see how much more you can get out of them. Learning from anti-patterns is a continuous and very rewarding activity for aspiring architects.

We have now achieved a foundational understanding of what anti-patterns are and how they can help us achieve greater mastery of Salesforce architecture. It only remains to summarize our progress before we dive into the deep end by looking at anti-patterns in the system architecture domain.

Phew, that was a lot of background. Thank you for sticking with it. You are now in a position to move forward with the book having understood the key facts about what anti-patterns are and how you can learn from them.

A few things to remember as you go into the next chapters are as follows:

• It is important not to simply think about anti-patterns as bad practice. Bad practice is obvious, but anti-patterns are nefarious. They masquerade as good solutions and can be persuasive in certain situations, even to experienced professionals. That’s also what makes them so fun to study.
• While we will be presenting a lot of individual anti-patterns in this book, it is important to draw out the parallels and meta-patterns that can be gleaned by comparing and contrasting them. We will do some of that work for you in the analysis sections that can be found throughout the book. You should also try to do this activity for yourself.
• While we are focusing on a Salesforce context, many of the patterns you will learn about are applicable to other platforms as well. Some are even classics that will be known by a great number of grizzled architects.

With that out of the way, we are now ready to look in detail at our first architecture domain: systems architecture.