The Professional ScrumMaster's Handbook: A collection of tips, tricks, and war stories to help the professional ScrumMaster break the chains of traditional organization and management

Complex adaptive systems are made up of varied interdependent elements whose relationships change as a result of experience. Complex systems adapt; as one variable emerges, others are affected. A simple example from biology is the Staph bacteria; in the 1940s, more than 99 percent of Staph bacteria were sensitive to penicillin. Today, Staph is resistant to penicillin; due to the introduction and exposure to the antibiotic, the bacteria evolved to survive.

In our world, requirements, technology, and people are intertwined; a change in one means a change in the other. When I was a kid, I loved to go to the movie theater to, and sometimes I would go just to play video games at the adjoining arcade (Ms. Pac-Man, Frogger, Pole Position, and Asteroids). My family couldn't afford a home system such as an Atari (and, having played at the arcade, I much preferred the arcade to an Atari home system anyway). Well, all that changed when Nintendo launched its Nintendo Entertainment System (NES). The kids next door got the system as a Christmas gift that year and next thing you know, we were over at their house many times per week after school playing Super Mario Brothers. Then along came the handhelds, Super NES, PlayStation, Sega to name a few, and now mobile phones and interactive play are all the rage. The advances in 16-bit and 32-bit consoles initially improved the user's experience at home with better graphics, competitive pressure resulted in more innovative home products from which a user could choose, and new technologies have allowed gaming companies to extend offerings into new arenas and find new markets. The technology, user needs and requirements, and developers in the gaming industry are all intertwined. A change in one necessitates a change in the others. Little did I realize, at that time, Mario and Luigi's importance to technology—much greater than just saving the princess!

Here's the deal: today's users change their minds about needs and desires minute to minute, developers continuously release new technologies and better versions of the old, and stakeholders and customers are never shy about saying how they feel about existing features. This emergence, that is, the evolution of new knowledge, needs, and desires about requirements and technologies during the course of a project, contributes to one heck of a complex situation. Complex projects cannot be adequately managed in a traditional approach, where all tasks are defined up front and then delegated to workers. In situations where the exact answer is not known, and must be discovered, an empirical process or evidence-based process is the best approach to use. (Take a look at the article A Leader's Framework for Decision Making, by David J. Snowden and Mary E. Boone, at Harvard Business Review). But it's more than just applying the right tool or technique to the matter; complex systems involve complex human interactions. While a complex project requires a process that allows for emergence, it also requires a leadership style that facilitates humans getting along together.

The following figure is the famous Stacey Matrix, adopted from Dr. Ralph Stacey, Professor of Management at the Business School of the University of Hertfordshire, who has spent many years exploring how the complexity sciences can be leveraged to understand organizations. You may remember hearing about his work during your ScrumMaster training. The Stacey Matrix looks at certainty and agreement, which correlate to technology and requirements, respectively, in our world. The farther from certainty and agreement a situation is, the more complex or even chaotic the situation; on the other hand, a simple situation is one with agreement and knowledge. Think about an easy project from your career. The requirements for the new feature were very clear and straightforward, and you wrote all the existing code a few weeks ago. It was easy for you to provide a time estimate since you figured that surprises weren't likely. This is a simple situation, rare in the software world. A complex scenario, on the other hand, is one in which you may not have been very familiar with the code, the customer often changed her mind, and the vendor team never met its deadlines. In this case, it was impossible for you to estimate beyond the tip of your nose as surprises (mostly bad) were guaranteed.

You'll notice that I overlaid the Scrum roles and control points onto the Stacey Matrix. ScrumMasters know that they do not control complex projects by acting as task masters and telling everyone what to do every day; rather, control is acquired by allowing a team to self manage through a series of sprints (or time-boxes) and by requiring the product owner to force rank the product backlog. Each role in Scrum has a specific and important set of responsibilities, each steeped in the idea of controlling chaos (in fact, Ken Schwaber's original Scrum website is called www.controlchaos.com). Many people ask me if there is a role called project manager in Scrum and the answer is a resounding no! The reason is that by giving control to only one person over what gets done, how it gets done, and why it gets done results in chaos and other bad effects. Three main responsibilities divided among three main roles creates checks and balances in our complex project systems; these three main Scrum roles, coupled with a simple process framework, control chaos. Each sprint, by focusing on a narrow selection of product backlog, an empowered team devises the appropriate technical solution bit by bit, thus bringing the project closer to the simple range sprint after sprint. If these controls are not protected, it is all too easy for the scale to unfavorably tip. Scrum is a very simple concept and extremely powerful once implemented (and yet rather difficult to do). You, my dear ScrumMaster, must keep that sprint to the simpler side of things by thwarting interruptions and helping the product owner keep the product backlog in order.

Even though Dr. Stacey's matrix and Snowden and Boone's article provide us with nice frameworks of thought to choose the right process for the situation, please do not oversimplify this information! Dr. Stacey didn't publish the matrix beyond the second edition of his book, and he's now up to six editions! What happened? Dr. Stacey stopped publishing the matrix because people oversimplified it, thinking that one can just choose a tool or technique depending on the classification of their project. Dr. Stacey warns us that it's not that simple, that, "what happens is the result of the interplay between the intentions and strategies of all involved and no one can control this interplay, hence the fundamental uncertainty and unpredictability of human life." This is another way of saying that Scrum doesn't fix the problems, people do; Scrum only exposes where the challenges lie. People must solve the problems exposed by Scrum and should craft different solutions for different scenarios. This book attempts to explore how the ScrumMaster may facilitate some of those challenges both inside and outside of the team.

Knowledge is the outcome of experience. In a traditional approach to managing projects, all the decisions are made at the beginning of the project. This has always been interesting to me because a person has the most knowledge about a project at the end of the project! Scrum puts urgency on teams so that they deliver value quickly; in doing so, knowledge about requirements and technology increases rapidly and team dynamics begin to emerge and stabilize. The sooner a team gets started, the better. When a team has knowledge its members can make better, well-informed decisions. Scrum teams delay decisions around lower-value requirements for a later point in time, as requirements in their fast-paced technology worlds are bound to change anyway.

There are four legs supporting the Scrum barstool that make it unique and effective in complex situations: prioritized product backlog, dedicated cross-functional team, time-boxes, and inspection and adaptation. These four entities form the necessary boundaries that allow us to call Scrum an empirical process. When a barstool loses a leg, it is imbalanced (ever try sitting on one?); even if one leg is slightly shorter than the others, the barstool is wobbly and annoying. The same idea applies with the Scrum concepts: when the team become lax on one or more boundaries, the project gets loosey-goosey; the experiment gets shaky as it loses its control points. For example, if for some reason the ScrumMaster allows interruptions during a sprint, the team runs the risk of not finishing anything by the sprint's end. Everyone must stop and start again—we call this chaos. Likewise, if the product owner does not rank features in the product backlog, and no thought is given to priority, the team may or may not deliver the right features to customers. A simple violation of these guiding principles can cause major downstream effects. If a team adheres to these four principles of Scrum, it will likely attain a higher degree of stability and knowledge.

Ken Schwaber and Jeff Sutherland designed these four "legs", or principles, of Scrum to enable empirical process control for software development projects (although the techniques are useful in just about any type of project). Scrum is an evidence-based way of progressing through a project or initiative. Without designated points in time around which people can provide feedback on features, like in waterfall projects, the team, management, and customers really have no idea what is being developed. Try to recount a status meeting for a traditionally managed project. What was that meeting like for you? I can say that, for me, it was often daunting. I would present a status report with a bunch of red, green, and yellow dots on it, some numbers and percentages of 'phase complete', and sometimes a list of risks. I can honestly say that as a traditional project manager, if I didn't truly know the status, I'd just put a yellow dot on the dashboard, reckoning I'd figure it out later. Or, even worse, I'd stay at work very late for a few nights ahead of the status meeting attempting to make the Gantt chart look something like reality. Scrum, with its emphasis on potentially shippable product increments each and every sprint leaves nowhere to hide. Sprint reviews replace status meetings; and the status is real—that is, the team shows real features that work! That is exciting and tangible! It's evidence, and we can work with evidence! Instead of a bunch of guesswork about what we think is the status of the project, in Scrum we can actually see, touch, and feel the status of the project by attending a sprint Review and interacting with the software or system as it exists at that point in time.

I recently watched a television program that featured George Cleverley and Company, a famous London shoemaker. Founded in 1958, Cleverley makes some of the most expensive handmade shoes in the world, shodding some of the world's most discriminating tootsies. In order to make such a beautiful product of unprecedented quality, the shoemakers must work very closely with the customer to understand their desires and requirements. After crafting precise shoe-lasts (forms around which the shoe is built) made from several measurements of a customer's foot (bunions and corns included), the shoemaker must then work with the customer's choices regarding various leathers, soles, dyes, laces, thistles, welts, and stitching in order to make the shoe that the wearer has envisioned and will love. It takes months for a customer's shoes to be finished, and at least three thousand U.S. dollars, but for 58 years customers have gladly paid and anxiously waited for the call that their shoes are ready. George Cleverley and Company involves its customers in the design and creation; there's simply no other way to build the perfect shoe. In reading about the craftsmanship, attention to detail, and customer satisfaction—the hallmarks of the Cleverley brand—I realized that the process in which Cleverley shoes are built is very Scrum-like: an iterative creation process where the customer is an essential part of the desired outcome. Even though the product backlog in this process is relatively fixed—that is, the order of the steps in shoemaking is fixed, the customer must be involved every step of the way.

In addition to the basics in complex adaptive systems theory, Scrum also has at its center five core values: commitment, focus, openness, respect, and courage. (Take a look at Agile Software Development with Scrum, by Ken Schwaber, Prentice Hall.) We will explore these in more detail in Chapter 8, Everyday Leadership for the ScrumMaster.

Teams commit to their goals for the sprint, product owners commit to ordering the product backlog, and ScrumMasters commit to removing obstacles along the way in order to steady the flow of product development. The Scrum team should do whatever is necessary in order to meet their goals, and it's important that they are empowered to do so.

Additionally, for a team to be able to complete its work, its members must be allowed to focus. The ScrumMaster does not allow changes in the sprint's commitment so that the team may keep its focus. When a team gives its full attention to the problem, its work is much more productive, predictable, and fulfilling.

As Scrum uses empirical process control to make progress through a project, it is essential that the results and experience of an experiment (that is, a sprint) are visible. For example, the sprint review provides visibility into how the product's features and capabilities have emerged, and the sprint retrospective engages the team members to discuss their personal and team successes and challenges experienced during the sprint. Once visibility exists, inspection and adaptation can occur. Again, evidence (information) is the heart of an empirical process; thus, openness is one of the five core values.

Respect is something that all humans should have for each other, but unfortunately it does not always exist. In order to be its best, a team's members need to respect for each other and, the knowledge that each brings to the table, experiences, working styles, and personalities, just to name a few. Respect doesn't come for free; it is earned. Scrum team members should be dedicated, cross-functional, empowered, and self-organizing—any team that is not provided this environment or structure will have a tough time achieving mutual trust and respect. We will discuss in subsequent chapters how a ScrumMaster can model (and thus create) trust and respect within the team.

It takes buckets of courage for a ScrumMaster to apply Scrum the way it was intended. Even though Scrum is very simple in design, people end up over-engineering its intended plainness. I often see management at various companies bring in Scrum as a way to do more with less (not always the case!), and they don't at the very least realize or acknowledge that the use of Scrum puts pressure on organizations to change. One of the primary responsibilities of the ScrumMaster is to help the organization identify its weaknesses so that it may improve. This takes courage. Sometimes, a team has to push back on the product owner when asked to take on too much during a sprint. It takes courage to say no to that sort of pressure. A product owner must have courage when communicating with other stakeholders about the reality of a project. These are just a few examples that require courage in order to not break the boundaries of empirical process or introduce more waste into the product or process.

An interesting tidbit: the Scrum core values sound a lot like the U.S. Army's seven values of loyalty, duty, respect, selfless service, honor, integrity, and personal courage. We might not find that too surprising as the grandfathers of Scrum have military backgrounds: Jeff Sutherland was a Top Gun fighter pilot in the U.S. Air Force, and Ken Schwaber attended the U.S. Merchant Marine Academy.

Lean software development is the result of the translation of lean manufacturing into the software and systems development space. Whether it falls underneath the Agile umbrella is hotly debated in the community, yet it really doesn't matter when it's all said and done. This is because the seven principles of lean are inherently reflected in Agile methods, but perhaps in different ways and with different lexicons or naming conventions.

The first principle of Lean is to eliminate waste. From unclear requirements, unused documentation, hand-offs, wait time, and so forth, lean practitioners seek to reduce or eliminate altogether these wastes from the process. The Scrum framework echoes this lean sentiment by providing the retrospective so that a team may discover and fix anything that's not working well. You'll often hear teams discussing subjects such as wasteful documentation, overbearing and/or manual processes, too many defects, and other such issues in the retrospective.

Lean thinking also stresses that everyone on the team gets the chance to learn. Scrum mirrors this by requiring that every sprint ends with a potentially shippable product increment. This quality threshold requires that team members code, integrate, and perform tests, of various sorts in order to learn what is defective, while the product owner learns about the emerging product. Additionally, members of a team work very closely together, which means that they often learn a little about the others' domain.

In any project, people know the most about the project at the end of the project. Scrum teams would rather make well-informed decisions; therefore, they do not make decisions about every requirement up front. This is a manifestation of the fourth value of Lean: decide as late as possible.

Lean says deliver fast; in Scrum we deliver at most every 30 days, while many teams deliver even faster than that. Lean says empower the team, and so does Scrum. Lean says that integrity should be built into the system; Scrum answers this by requiring a team to define done with a customer. Finally, Lean guides us to see the whole—how the entire value stream, or chain of events leading to customer value, operates. Any bottlenecks should be removed immediately, and teams should be staffed so that they can complete finished product increments. Scrum reflects this by guiding us to create dedicated, cross-functional teams that conduct retrospectives. Such retrospectives help us unearth bottlenecks (or obstacles as called in Scrum) so that they can be eliminated.

The Scrum team includes the product owner, ScrumMaster, and the team members, whereas the Scrum delivery team is a subset made up of only the technical team members. The entire Scrum team huddles around a problem (a requirement from an ordered list known as the Product Backlog) and innovates solutions. Scrum teams should be five to nine team members, dedicated to the life of the project (and perhaps beyond), cross-functional, empowered, and self-organizing. Scrum teams plan, estimate, and commit to their work, rather than a manager performing these functions for them. The end goal of the team is to deliver a potentially shippable product increment that meets an agreed-upon Definition of Done each and every sprint. The George Cleverly shoe craftsmen would be considered Scrum delivery team members while the craftsmen, customer, and shoe design consultant together would comprise the overall Scrum team (it takes everyone's participation to make the right shoe).

The product owner is responsible for the product's success. In other words, while the team is responsible for delivering a quality solution, the product owner is responsible for knowing his market and user needs well enough to guide the team toward a marketable release sprint after sprint. There are different types of product owners in different types of projects, but regardless of the technical situation or the desired outcome, there should be one (and only one) product owner who makes final decisions about the direction of the product and the order in which features should be developed. The product backlog, or list of items to be completed by the Scrum team, is ordered by the product owner so as to reflect his most valuable requests or changes for the product in development. The product owner, since he is representing the "what" and "why" of the system, should be available to the team to have regular dialogs about the requirements in the product backlog; additionally, the product owner must make the product vision clear to everyone on the team and regularly maintain the product backlog in keeping with the product vision. The product owner always keeps the next set of product backlog items in a ready state so that the team always has work in the queue for the next sprint. The George Cleverley shoe customer is the 'purest' product owner a team can get—the actual person paying for the product!

The ScrumMaster safeguards the process. He/she understands the reasons behind and for an empirical process, and does his or her best to keep product development flowing as smoothly as possible. This "servant leader" (take a look at Servant Leadership, by Robert K. Greenleaf, Paulist PR) protects team members from interruptions in order to keep them focused on their sprint commitments, as well as helps the product owner get the product backlog in order if he or she does not understand how to do so. Additionally, ScrumMasters facilitate all Scrum meetings, ensuring that everyone on the team understands the goals and that they share a commitment together as a true team and not just as a collection of individuals. ScrumMasters remove obstacles that prevent a steady flow of high-value features; many times, the obstacles are organizational in nature.

Think of the ScrumMaster as the Switzerland of the Scrum process—remaining neutral, helping both business stakeholders and development, interacting at the right times to create the most important and most valuable features first.

During sprint planning, the second time-boxed event, the product owner and the team discuss the highest priority items in the product backlog and brainstorm a plan to implement those items. The set of chosen product backlog items and their subsequent tasks collectively is referred to as the team's sprint backlog.

The sprint planning meeting is time-boxed to eight hours for a 30-day sprint, reduced proportionally for shorter sprints (for a two-week sprint, for example, one may reduce that time to a maximum of four hours). The meeting is comprised of two parts. The first segment of the meeting is driven by the product owner who presents the most important product backlog items—along with any helpful drawings, models, and mockups, for example—and clarifies questions from the development team about what he/she wants and why he/she wants it. The second segment of the meeting is driven by the Scrum delivery team who work together to brainstorm approaches and eventually agree on a plan. It's at the start of this second segment that the sprint actually begins.

Of course, teams are always searching for ways to make planning faster and more efficient. You will likely see sprint planning take less time over a number of sprints for a team whose members have remained consistent.

The result of sprint planning is a sprint backlog that is comprised of selected product backlog items for the sprint, along with the correlating tasks identified by the team in the second segment of sprint planning. We'll discuss many details and ideas for effective sprint planning meetings in Chapter 3, Sprint Planning – Fine-tune the Sprint Commitment

In the daily scrum meeting, the third time box of Scrum, team members make their progress visible so that they can inspect and adapt toward meeting their goals, sort of like a mini-Deming (PDCA) cycle every day! The meeting is held at the same time and in the same place, decided upon by the team. Even though a team makes its best attempt at planning for a sprint, things can change in flight. Tom finishes a task late, Beth finishes early, Ashish is stuck. In this 15-minute meeting, team members discuss what they did since yesterday's meeting, what they plan to do by tomorrow's meeting, and to mention any obstacles that may be in their way. The role of the ScrumMaster in the daily scrum is to facilitate, keep the conversation from delving into problem solving, and record any obstacles that team members feel they cannot fix for themselves. The ScrumMaster will attempt to remove said obstacles after the meeting. The daily scrum meeting represents inspection and adaptation at a daily level. The Scrum delivery team members, product owner, and ScrumMaster are participants in the meeting. Anyone else is welcome to attend but only as observers.

The sprint review, the fourth time-boxed event of Scrum, provides the opportunity for stakeholders to give feedback about the emerging product in a collaborative setting. In this meeting, the team, product owner, ScrumMaster, and any interested stakeholders meet to review the features and talk about how the product is shaping up, which features may need to change, and perhaps discuss new ideas to add to the product backlog. It is common for a ScrumMaster to summarize the events of the sprint, any major obstacles that the team ran into, decisions that were made in-flight with the product owner's help, and so on, and of course the team should always demo what they've accomplished by the sprint's end. This meeting is time-boxed to four hours (for a 30-day sprint).

During the final sprint meeting— the sprint retrospective—team members discuss the events of the sprint, identify what worked well for them, what didn't work so well, and take on action items for any changes that they'd like to make for the next sprint. The ScrumMaster will take on any actions that the team does not feel it can handle; likely broader organizational issues. The ScrumMaster reports progress to the team regarding these obstacles in subsequent sprints. This meeting is time-boxed to three hours. We will dive into more details about sprint reviews and retrospectives in Chapter 5, The End? Improving Product and Process One Bite at a Time.

The final time box in Scrum is release planning, an optional event, in which Scrum teams plan for long-term initiatives comprised of multiple sprints' worth of work. The product owner and team discuss product backlog items, dependencies, risks, schedule, and capacity, among other topics, in this meeting to settle on a forecast for upcoming work. Since the product backlog can be infinite, release planning helps the team and product owner understand what may be possible given a release deadline. This subset of the product backlog is called the release backlog and is a valuable output of the release planning meeting. Release planning should be time-boxed but the time required depends upon the scale of the program, the number of teams, team distribution, and how well the product backlog is prepared. Release planning meetings are often held at the beginning of the project, and teams will meet with their product owners throughout the project to re-plan based on emergent knowledge. It is not unheard of, however, for teams to postpone planning a release until they have a few sprints of work under their belts, as the experience and knowledge they gain in a few sprints results in greater confidence in a longer-term release plan (more details in Chapter 2, Release Planning – Tuning Product Development). Either approach is acceptable and depends upon the needs of the organization.

Ken Schwaber once said to me, "I don't know why people make this such a long, drawn-out event. It isn't really that hard." And with that, we completed our release planning in 15 minutes.

The product backlog is the product owner's 'wish list'. Anything and everything that they (and other stakeholders) think they might want in the product goes in this list. It could be infinite as there are always new ideas about how to extend a product's features. The product owner maintains the product backlog, although other stakeholders (including the team) should have visibility of and the ability to suggest new items for the list.

The product owner prioritizes the product backlog, listing the most important or most valuable items first. That is, there aren't 10 critical items at the top of the backlog with equal value; rather, there are 10 critical items that are ranked according to their priority or urgency, and they appear at the top of the product backlog, one after another. This is because items at the top are next in the queue to be implemented. Once a team selects items for a sprint (or iteration), those items and their priorities are locked; however, priorities and details for any not-started work may change at any time. Through this mechanism, teams are able to focus on this sprint's work while the product owner retains maximum flexibility in ordering the next sprint's work.

Product owners have many ways of evaluating and thus prioritizing their lists. They may also attribute product backlog items with additional information such as improves brand recognition, allows scaling, infrastructure, contracts greater than 10,000 dollars, and so forth. Attributes are unique to a particular company and a particular product and help the product owner to keep the list in the proper order.

Owned by the team, the sprint backlog reflects the product backlog items that the team committed to in sprint planning, as well as the subsequent tasks and reminders. Team members update it every day to reflect how many hours remain on his or her tasks; team members may also remove tasks, add tasks, or change tasks as the sprint is underway.

The product increment is a set of features, user stories, or other deliverables completed by the team in the sprint. The product increment should be potentially shippable—that is, of high enough quality to give to users. The product owner is responsible for accepting the product increment during each sprint, according to the agreed-upon Definition of Done and acceptance criteria for each sprint deliverable. Without a product increment, the product owner and other stakeholders have no way to inspect and adapt the product.

A subset of the product backlog that has been identified for a particular release is called the release backlog. Even though release backlog can be defined up front, the product owner may remove items, exchange items, or negotiate scope depth for some items as he/she considers scope, time, and cost throughout the duration of the project. Therefore, the release backlog should be updated throughout the project. Chapter 2, Release Planning – Tuning Product Development, provides more detailed information about product backlogs, release burndowns, and backlogs.

The release burndown chart displays how much work remains in the release backlog at the end of each sprint. This provides the product owner with important information so that he/she may make well-informed decisions about scope, cost, and time. In the following diagram, you can see that the amount of work remaining at the end of each sprint is more than the work planned:

During a particular sprint, if everyone on the team updates the sprint backlog with the number of remaining hours per task every day, then the team can see if they will be able to burn down the number of task hours by the end of the sprint. In the following figure, you can see that the team did not complete all the tasks it had identified in sprint planning; approximately 50 hours remain.

This burndown concept is very important because, once set, the sprint end date does not change. Coupled with a daily Scrum, the sprint backlog and burn down chart can help teams visualize when they might be getting off track and turn the conversation to focus on what to do about a given situation. The sprint burndown "burns down" hours over days of the sprint, while the release burndown looks at units of work (often referred to as points) for a release, or number of sprints. Chapter 3, Sprint Planning – Fine-Tune the Sprint Commitment and Chapter 5, The End? Improving Product and Process One Bite at a Time, provide more details about burndowns and backlogs.