The more I work with companies that are struggling with Scrum, the more I’m starting to believe that ‘hybrid’ Scrum adoptions, where people pick and choose which Scrum practices to follow and which to ignore, invariably lead to failure.

Whoa! you say… Wait a minute! Agile is about doing what is right, not following a process! It says so in the Agile Manifesto! “Individuals and interactions over processes and tools!”

Listen up, Sparky! The Agile Manifesto has to be taken in context, not in isolation. We value processes and tools unless they interfere with individuals and interactions. That is not the same thing as dismissing processes and tools because we value individuals and interactions. Remember, there is value in the items on the right. Too many teams use the ‘left over right’ emphasis to justify haphazard adoption of Scrum, resulting in The Dreaded Plague Known As Scrum-But. As in, “We’re doing Scrum, but we’ll finish testing at the end of the release,” or “We’re doing Scrum, but we have a weekly standup because it’s more efficient.”

How do you Scrum-But? Let me count the ways…. I think the variations are infinite, but here’s some I’ve run into in just the past three months:

• No sprint retrospectives
• No iterations! (the team was following a quarterly stage-gate plan under a deterministic methodology)
• No assigned Scrum Master (people would take turns being called ‘Scrum Master’ and perhaps running specific meetings on some indeterminate schedule), leading to a Product Owner who came in and changed the sprint backlog at will because no one owned ensuring adherence to the Scrum process
• No sprint reviews, with the team deciding if a backlog item was done or not as a matter of opinion/consensus
• No sprint burndown chart… of any type… and no tracking of velocity (“What’s velocity?”)
• No self-assignment of tasks and a ‘Scrum Master’ who was creating the sprint backlog and assigning tasks to team members
• No defined product backlog and a Product Owner who abdicated his role by shoving all backlog work onto the team (the team would spend the first several days of a two-week sprint creating the sprint backlog with time estimates)
• etc.

One thing that all of these teams had in common: they were all failing to some extent, and some were failing spectacularly. Many blamed Scrum, saying “Scrum doesn’t work!” As if blaming a process for your failure when you’re not following the process somehow makes the failure okay, because it’s beyond your control. It’s Scrum’s fault!

But you don’t understand! We can’t change completely over to Scrum all at once. The team will never accept it! We have to ease our way into it! It requires too much common sense! And besides, our management would never support adopting full-fledged Scrum. We have to do it this way. Be reasonable!

Horsefeathers! (This is a family blog.) Whenever I hear this, what I’m really hearing is We don’t understand what Scrum is, how to do it, or why the minimal Scrum processes and practices are mandatory, because we don’t understand the why. And I hear the undercurrent thought, too: We’re afraid. We’re afraid we’ll fail because we’re not sure how to do it, and we don’t trust ourselves or our fellow team members.

What is Scrum, anyway? Is it just another way of doing your job? Of project management? Of organizing and assigning work? Doesn’t Agility really mean picking and choosing what is best for your organization?

I could (and should) do an entire blogpost on the meaning of Agility, but here it is in a nutshell: Agility is the ability to respond effectively to change. That’s it. Agility doesn’t mean you don’t have to plan, or you don’t have to follow a process, or you don’t have to be disciplined. Being Agile requires all of these, just as anything worthwhile… relationships, career success, success in sports, proficiency in music, etc… requires thought, effort, and discipline.

What is Scrum? It’s three roles, four meetings, and two levels of commitment. What are you saying about yourself or your organization if you’re not willing to try to follow a minimal process with three roles, four meetings, and two levels of commitment? Why is picking and choosing from those minimal processes and practices so devastating to effectiveness? The first reason is because if you haven’t run Scrum for a while and gotten really good at it, you almost certainly have no idea of the why behind each specific process and practice. You may read a book, or take a class, but you won’t get it, just as you could read a book about how to ride a bike but until you get out there and try, and fall, and get up and try again and again, you won’t get it. I understand the fear of change, and the reluctance to go outside one’s comfort zone, but success requires the courage to try. Yes, “a ship in harbor is safe, but that is not what ships are built for.” If you’re not willing to step up, to be serious about developing software, then perhaps this is not the career area for you.

Let’s look at the why behind some of the pick and choose adoption decisions:

• Why do we need to hold sprint reviews after every sprint? Because working software, not a Gantt Chart, not a checked-off To-Do list, and certainly not written code, is ultimately the only true measure of progress.
• Why do we need a sprint retrospective after every sprint? Because unless we take the time to examine the effectiveness of our engineering processes, we will never improve them, and we will never get better at delivering software if we don’t change how we do it.
• Why do we need a sprint burndown chart when we’ll know if we’re done at the end of the sprint anyway? Because sprints, like projects, don’t fail all at once at the very end; they fail a little each and every day. The burndown chart gives us a mechanism to detect difficulties early while we can still take effective action to correct the problem and prevent the failure.
• Why can’t we let the Scrum Master act as a project manager or team lead and assign task to individuals? Because we are substituting the judgment of one brain (the Scrum Master’s) for the collective judgment of a lot of brains (the teams’), and regardless of how smart the Scrum Master is, there is no way that he or she knows more about every single aspect of the work than the entire team. Additionally, the team will only grow if given the opportunity, and solving their problems eliminates the opportunity for growth.

And so on. Leaving out these and other processes and practices breaks the paradigm and removes the opportunity for continuous improvement of the product and the process. Why would you want to do that? Scrum is three roles, four meetings, and two levels of commitment, but it’s more. Scrum is an organizational change metaprocess disguised as a project management process wrapper. Scrum exposes your problems, if you have the courage to follow the process despite what you may discover about your organization, your team, and yourself.

There’s a scene in The Matrix, a movie about choosing which path in life to take disguised as science fiction, where the protagonist is offered a choice: take the Blue Pill and go on in the same way, or take the Red Pill and learn the truth about what is really happening in order to do something about it. Scrum is that choice. You can choose to fail, to continue developing software the way you’re currently going about it, to get the same results you’re getting. As W. Edward Deming pointed out, “Failure is an option.” Or, you can take the Red Pill. You can choose a metaprocess that exposes your failures, and then you can choose to fix them, to do things differently, to do things better. Of course, you can choose not to choose, and that in itself is a choice. Saying you’re adopting Scrum when you’re picking and choosing from the minimal processes in order to avoid the pain of change is just fooling yourself.

Scrum done right is the Red Pill. The choice is yours. Choose wisely.

…and start solving them. Sounds great, but what does it mean? What’s the difference between managing a problem and solving it?

I recently held a workshop on using Scrum to drive process improvement at CIISA 2009, held in Guadalajara, Mexico, where I focused on using Scrum as a process improvement process to find problems with a development organization’s processes and practices and then using the A3 Problem Solving process¹ to drill down to the root causes and eliminate them. Let me give you a brief overview on what I covered.

Every software development organization can deliver some functionality, at some level of quality, in some amount of time. But can they deliver a specific, committed amount of functionality, at a high level of quality, in a specific amount of time? That is what we are asking for at every sprint planning meeting. Do we get it? Usually not, when we first transition to Scrum. Failing to meet a commitment is not unique to Scrum; the vast majority of traditionally-managed projects fail to deliver the desired functionality within the established schedule and cost constraints. One advantage of Scrum is that you fail faster, while you still have time to do something about it.

Failing to deliver isn’t bad… if you seize the opportunity to improve. Failures happen for a reason. What was the reason? Why did you fail? This next statement may seem tautological in its obviousness, but true wisdom is often found in simple profundity. You failed because a problem exists that prevented you from succeeding. You won’t stop failing until you recognize the problem, understand the root causes, and then make the changes required to eliminate the root causes and solve the problem.

Is this how most companies handle problems? I don’t think so. It’s human nature to only want to see the good things. Twenty-five hundred years ago, the Athenian playwright Sophocles wrote that “No one loves the messenger who brings bad news” to reflect the then-common practice of an angry ruler killing the bearer of bad tidings. Shakespeare’s “Don’t shoot the messenger” has become a common metaphor, advising us to not blame the person for the problem. We see this around us every day; someone yelling at testers when they find a lot of bugs, or rejecting high estimates and demanding ‘more reasonable’ ones, etc. Perhaps we need to rethink our approach to problems that are expressed as bad news.

At Toyota, honesty is prized. Managers are taught to present bad news first, and no one is admonished for doing so. To the contrary, managers are severely admonished if they don’t bring bad news to their superiors. Similarly, Jim Collins writes about the Stockdale Paradox in “Good to Great.” Companies that excel do so because they maintain faith in their ability to prevail in the end while simultaneously confronting the brutal facts of their current reality, whatever those facts are. They embrace the truth, no matter how awful it may be, because they understand problems must be acknowledged before they can be solved.

Ken Schwaber, one of the creators of Scrum, says “Scrum doesn’t solve your problems, it exposes your problems.” Scrum isn’t a Silver Bullet. Your organization needs to solve its problems if it wants to get better. Managing a problem is kicking the can down the road, working around the problem, putting it off until it can no longer be avoided. All too often that is exactly what we do.

For instance, let’s say that one day last week you walked out to your car and noticed the front left tire was low on air, so you went by the local gas station on the way to work and filled it up. A few days later, you noticed it was low again, so you filled it up again. You managed the problem. Now the weekend is here, and you’ve scheduled a round of golf with your buddies in the morning followed by an afternoon at the stadium watching your local team. Now you have a choice: do you change plans so you can go down to the gas station and get the leak found and fixed, or not? Sure, you can keep topping it up every couple of days, but we all know what eventually happens to a leaky tire. It fails at the worst possible time, perhaps a blow-out on the freeway, or else we walk out after a late night at the office to find ourselves staring at a flat. Problems that aren’t solved only get worse, and the solution becomes more onerous.

Solving problems is a choice; it is up to your organization to solve them… or not. You have to have the courage to exercise integrity at the moment of choice², to make the decision to solve problems because not making a decision is making a decision. Actively decide to solve your problems.

Contact me if you’d like an A3 Problem Report template, along with examples and instructions.

______
¹Tom Poppendieck, Gabrielle Benefield, and Henrik Kniberg led an excellent workshop on value stream analysis and problem solving using the A3 Problem Solving process at the Agile2009 conference… thanks!

²The Seven Habits of Highly Effective People, Covey, Stephen

Technorati: gu7dh42snr

Many teams moving to Scrum have questions about the Product Owner position. Is the Product Owner a member of the Scrum team? What role does the Product Owner play in the day-to-day life of a Scrum project? How do we map current functional roles to Scrum roles, specifically with regard to the Product Owner? Who should we select as our Product Owner?

Let me start by saying the Product Owner is perhaps the most important role in Scrum… something you don’t often hear from Scrum folks. The Scrum process defines the Product Owner as being the person responsible for the team’s return on investment, i.e., the Product Owner will be judged by whether the project’s outcome justifies its cost. Another, more direct way of saying this is to identify the Product Owner as “The Single Wring-able Neck,” or the person whose head is on the figurative chopping block if the project fails. Interestingly, the Project Management Institute has a similar definition for project managers: the person assigned by the performing organization to achieve the project objectives (PMBOK Guide, 4th Edition, p13). Therefore, based upon both the Scrum and PMI definition, the Product Owner responsibilities are equivalent to those of a project manager. This is one reason why I have found knowledge and competence in project management to be a key ability of a successful Product Owner. Selecting a Product Owner therefore naturally starts with identifying candidates who have that knowledge.

There are other skills and abilities required, including sufficient technical knowledge to understand the problem domain and the technical aspects of proposed solutions. A successful Product Owner doesn’t need to be the best software developer on the team, but he does need to be able to understand the technical decisions well enough to know whether they make sense.  Eliminate candidates who lack sufficient technical ability.

Be especially careful not to view the Product Owner as the project's ‘driver.’ Scrum is about empowered, self-managing teams which are led (pulled) rather than driven (pushed). Scrum means never having to be driven. Candidates who can’t or won’t embrace the servant-manager philosophy, or who insist on directing the team should be disqualified. Nothing will cripple your Scrum implementation more than a de jure Product Owner who sees himself as a de facto team manager.

By now, you should have narrowed the candidate list to those who have demonstrable technical, project management, and interpersonal skills. Who among the remaining candidates has the ability to understand the customer, the market, and the business? Are there candidates with entrepreneurial experience? Owning a business, starting a business, or working in a leadership position at a startup where everyone wears a multitude of hats and understands making money is the true test of whether or not the customer is satisfied is invaluable. People with this experience understand what really matters because they’ve lived it. People who have had experience in customer support, QA/test, or sales and marketing at larger companies may also have an understanding of the customer.

Now you should be down to the final few candidates. I like the Toyota Production System concept of a ‘Chief Engineer’ with extensive technical, project management, and business knowledge who leads the team to successful project completion. We’re talking about candidates with a software development background, successful project management experience, who have dealt effectively with the customer and understand business realities, and with the skills and experience to successfully act as a proxy for the customer and other stakeholders. Which of the remaining candidates most closely matches this description? If there is no one, have any of the candidates shown promise that they can develop to this level? If the answer is still “No,” then you may want to hire someone with the necessary talents and skills to fill the position.

 A question was posed on one of the Scrum discussion forums recently about changing the sprint backlog during a sprint. The scenario was as follows: the sprint has been running for 2 days when the Product Owner comes to the daily standup and wants to replace a committed sprint backlog item with one of equal size in the product backlog. What should the Scrum Master do?

As this was a real question posed to the forum, I wondered how often the Product Owner comes to the team two days into a sprint to rearrange the sprint backlog? What a stinky Scrum smell this is.

One of the major hindrances to getting something done is the Tyranny of the Urgent. The reason that Scrum keeps the committed sprint backlog inviolate is to provide the team the ability to keep their heads down for the sprint instead of constantly being whipsawed by context switches forced on behalf of the crisis du jour. If the level of uncertainty is very high in an organization, the proper response isn't to relax Scrum rules by allowing changing the sprint backlog during the sprint, it is to shorten sprint duration so the organization (and the Product Owner) know that, at most, they'll have to wait for no more than two iterations to get a specific item.

In my experience, the importance of changing the backlog during a sprint is almost always greatly exaggerated. (The 'importance' is usually due to someone outside the team dropping the ball on a commitment, and trying to cover it up by breaking the Scrum process.) Scrum has a mechanism for ascertaining if it should be done, however, and that is by forcing the Product Owner to make the call on whether or not to abandon the current sprint and replan with the updated backlog. Yes, this will cost up to a team-day. But following the Scrum process is exposing your problem, not hiding it, and there is a problem here. Are you going to address it, or sweep it under the rug by enabling bad behavior?

I know, some people will wonder what the big deal is. Why not just go along to get along? My response is to reiterate that Scrum is as much about improving the organization as it is organizing project work. Why waste a great opportunity for improvement? What I like about this particular scenario is how it shows whether you truly understand that Scrum is really a tool for improving the organization instead of just another way to manage a project or set up the workflow. Sure, you can go along to get along by swapping out backlog items and getting the Product Owner off your back, just as you can put a penny under a blown fuse and get the lights back on, but you haven't solved the problem. In fact, you've made it more likely to blow up.

Others will wonder if this somehow violates the Agile principle of responding to change over valuing a process. I'm all for responding to change. What I'm against is abandoning the Scrum process, and I'm suggesting that there is a very good reason for the few simple rules that Scrum has. One of those rules is that the sprint backlog with the corresponding commitment is locked at the moment commitment is obtained and the sprint starts. Scrum allows for changes to the sprint backlog during an iteration; abandoning the sprint and replanning and restarting a new sprint. As an aside, if you're following Scrum you shouldn't be dropping items from the sprint backlog during the sprint either; that is just a way of hiding what is truly happening. Instead, during the retrospective the team should discuss why committed sprint backlog items couldn't or wouldn't be completed during that sprint, or why the team undercommitted (if new items are continually being dragged into the sprint backlog), and then brainstorms to arrive at a solution to this problem.

Stephen Covey, of Seven Habits fame, says that "courage is integrity at the moment of choice." Scrum, or any other way of doing things, won't work if the people involved lack courage, if the prevailing philosophy is "Can't we all just get along?" instead of "Do the right thing." The Scrum Master needs to demonstrate some integrity in this situation and require the Product Owner to follow the Scrum process. Either abandon the sprint or leave the backlog alone. Then, during the retrospective, a discussion on why the backlog needed to be changed should be held with the goal of trying to figure out what caused this (genuine unforeseeable emergency, bad PO planning, someone dropped the ball on a commitment, etc.) and how to prevent this in the future. Otherwise, expect constant pressure to change the sprint backlog.

Is Scrum really only valuable to folks who care about agility (or Agility)?

Let's say I'm running a project with defined requirements, fixed scope, a fixed schedule (firm completion/release date), and fixed resources. What advantages does Scrum offer over other project management methodologies?

How about the ability to maximize team efficiency? So, my requirements are clearly defined and I don't expect to change my product backlog based upon feedback from sprint reviews. So what? I still get the advantage of a pull system for work (scrum team members self-assign work efficiently instead of waiting for a manager or a Gantt chart to tell them what to do). I get the advantage of keeping the team from multi-tasking and other work-robbing interruptions. I get the advantages of acceptance criteria and the Definition of Done. I get the advantage of clearly knowing my status at any given time without a lot of overhead. I get the advantage of team and workflow process refinement/improvement via retrospectives. I get a simple system that lets me track work and predict when the project will be done fairly accurately. I even get the ability to reassure my customers and stakeholders that I am on track because I can show progress at regular intervals. In short, I get all of these advantages without any disadvantages... and I get a much easier project management framework to boot. The big bonus here is I get to show folks how an empowered team using a pull system can be much more efficient than the old command-and-control model while being easier to manage because they mostly manage themselves.

Sure, I can get some of these benefits of Scrum without running Scrum... but all of them? As easy as I can by just adopting Scrum?

Think what it would do to most organizations who don't value agility (the ability to react to change), to see the other benefits of Scrum and then realize they can get all this and the ability to course-correct without the pain and waste of throwing a lot of suddenly-obsolete work away. Being Agile means you can change but you don’t have to. Do what makes sense for your organization.

Many people dismiss the idea of trying Scrum because their organization is more interested in predictability than agility, not realizing that Scrum allows both. If whatever process you’re following now isn’t making you happy, maybe you should reconsider Scrum for all of these other reasons.

Welcome to the first post on my new software development blog!

Let me tell you a little about myself. I'm an experienced software developer, tester, program and project manager, QA manager, and development team manager, with over two decades of experience in high tech. I've worked at small startups and the world's largest software company. I've written device drivers, OS portability layers, libraries, utilities, and UI components, for environments including CP/M-80, MS-DOS, Windows and Windows CE, AmigaDOS, MacOS, PalmOS, Unix, VAX VMS, and IBM MVS/TSO. I've worked on development tools, desktop applications, and platforms. I've managed developers, testers, program and project managers, and entire development teams. And I've had the privilege of working with some very smart people at very successful companies... and with some very smart people at very unsuccessful companies.

I've learned that failure can be a great learning opportunity, and that the opportunities to turn failure into success are really within our control if we're willing to face the brutal facts, recognizing those signs of disaster that are so obvious in hindsight and then devising a solution while there is still time. Agile retrospectives are a great tool for this, as are other tools and techniques.

Many of my posts will be about what I see, and have seen, along with some general observations and perhaps answers to questions. Feel free to leave comments, or contact me through the blog or via Construx's website.