Software Best Practices

 

The agile community has heaped a lot of scorn on sequential development (often pilloried as "waterfall" development, though to my knowledge no one's used that term seriously in 10-15 yrs). But NO software method is ALWAYS wrong. It follows that there would seem to be situations where some form of waterfall is still the right choice. Anyone want to speculate on what those cases would be? Here's my first cut at a list:

1. Short projects. This is really like an incremental project with just one or two increments. Simple enough to get your head around, and if you go back, you're not going back far.
2. Projects where the need for predictability massively trumps the need for flexibility
3. Projects with a heavy regulatory burden, that require many additional parties to approve specifications (this is probably one of several instances of #2)
4. Projects where the requirements are relatively stable and will repay deeper exploration up front (as opposed to projects where the requirements are fluid enough that they can only be finalized by iterating on prototypes and code)

What other things might incline you to adopt or recommend a more sequential approach on a project?

--

Steve Lane

I completely disagree. In fact if you look carefully at any sucessful waterfall project you will probably find some measure of iterative development going on behind the scenes.

slane:

Short projects. This is really like an incremental project with just one or two increments. Simple enough to get your head around, and if you go back, you're not going back far.

But how short is short? Also short projects tend to need to be more agile in that unless what your doing is almost trivial you need to prototype and get feedback very quickly.

slane:

Projects where the need for predictability massively trumps the need for flexibility

I'd say almost the exact opposite - waterfall projects are the least predicatable and suffer from worse cost and time overruns than one run using some iterative and incremental process.

slane:

Projects with a heavy regulatory burden, that require many additional parties to approve specifications (this is probably one of several instances of #2)

Depends on who is doing the regulation. The US military in fact changed from specifying that projects must be run in a waterfall way to a more iterative way because of the massive failure of projects to deliver on time and on budget. And even where there is regulation it is possible to work in an iterative/agile way

slane:

Projects where the requirements are relatively stable and will repay deeper exploration up front (as opposed to projects where the requirements are fluid enough that they can only be finalized by iterating on prototypes and code)

I've never seen one of these rare projects. Do they exist?  

 

Hi David:

 I hear that you disagree -- but I'm not sure with which part :-) My post doesn't contain too many assertions -- it was more of a thought-experiment. About the only real assertion is "waterfall isn't always wrong." And I take it that you do disagree with that -- and by implication the idea that a sequential project could be done well, or rightly.

Of course, as often the case with discussions like this, some of our abstractions are straw men. No project is ever really run as "pure" anything. Even the most sequential project has some flexibility built into it. Even the most rabidly iterative project does at least some upfront planning (I hope). So it might be more fair for me to start by defining some terms.

Let's define a sequential project as "a project that tries to do a significant amount of discovery and design up front, makes the best plan possible based on that up front work, and then tries as far as possible to stick to that plan during construction." That's perhaps a less charged definition than "waterfall". Such a project could still be incremental, in the sense of doing and testing the work in chunks. I think it would be minimally iterative, meaning it'd be intolerant of significant change between increments.

 So to the extent that's a more nuanced definition, and I threw a red herring with the charged "waterfall" term, sorry for the rhetorical pit trap. :->

If we define a sequential project as I did above, and add a flavor of relative intolerance for change ("we value following a plan over responding to change"), that' s my question -- when is a more sequential approach (majority of planning up front, relative intolerance to change) the right one?

 

-- Steve 

My regulatory friends all use sequential development models. Many of my agile friends use lots of things from sequential development. They still have product road maps (sequential), release plans (sequential), and their product management groups are making promises to clients that cause them not to deviate too far from pre-planned requirements.

Let's face facts, time itself is sequential. The question is when are you going to do the detailed work on requirement/function/feature/story 'X'. Many of the sequential models today are doing just enough of that detail work to get through a stage gate and differing much of it until later in the project. And with schedule fixed sequential projects, they are doing incremental development but with more requirements discovery than one might see in a pure agile project.

It seems to me that the techniques are blending together to form some sort of sequenagile thing.

David Harper:

slane:

Projects where the requirements are relatively stable and will repay deeper exploration up front (as opposed to projects where the requirements are fluid enough that they can only be finalized by iterating on prototypes and code)

I've never seen one of these rare projects. Do they exist?  

 

May I suggest the famous C3 project? Or any other project where your task is to replace an old in-house system.

I'd agree with a lot of this. I think when I was referring to a project I was meaning a small thing with a defined end goal. But of course that end goal is set before hand. I'm not sure that I'd put product road maps and release plans in a development tool box? Maybe we are just using different terminology.

One question? Where does sequential and Agile meet? You do all of the same type of tasks - requirements, design etc in Agile - just in shorter cycles.

 As for replacing legacy systems - the question there is why are you having to do it? Normally it is because the requirements have changed. So I don't see that the waterfall model there is a good thing either!

I have been on replacing legacy systems projects because one of the platforms used was end-of-lifed. Yes, they did add some new requirements but they were pretty well known.

I think it is a great question to ask what is a "development project" that a "development tool box" should address. One of the things that many of our developer clients seem to complain about is that there is no road map or release plans. "How do we get management to settle down and tell us where they are going with this <insert product>?" Are agile approaches more helpful here in that they don't require a road map? (We will do what think is best today). Somehow it all has to flow together.

We are saying, somewhat tongue-in-cheek/somewhat serious, that "agile" today has come to mean "modern software development practices that work." So a group doing staged delivery (like FDD) can claim to be agile as those doing evolutionary prototyping (like XP). The real litmus tests seem to be more around buffering requirement into fixed duration increments and team empowerment.

 

David,

 I think the waterfall model is applicable when the requirements are well known even if they changed in the past. If you are replacing the old legacy system there is a reason for it, so you know why you do it and what needs to be done before you start the project. I think Earl was also refering to this point in his post.

 I think David puts it very well. ALL software projects do the same core tasks in some way -- requirements, design, construction, verification, refinement. One of the key distinctions among lifecycles is how they spread those activities out. You can even narrow it further and suggest that one of the key distinctions is how much requirements and design they try to do up front, vs. how much they spread those activities across the project. In a sequential project, the answer might be "as much as possible" (NB: not "all"!). On an agile project, the answer might be "as little as necessary" (NB: not "none"!). Between those there's a continuum. Generally speaking, the more fluid you expect the requirement to be, the more likely it's a good idea to slide toward the iterative end. The more stable the requirements, the more it becomes possible (though not always necessary) to do more up-front definition.

 I hadn't thought of legacy systems -- that's just the kind of example I was fishing for!

 Another reason you might do more up front relates to what I've heard Construx call the intellectual profile of a project -- is the project heavy on requirements (new domain, complex business rules)? Heavy on architecture (needs high scalability, integrates lots of components etc)? Heavy on construction issues? If the project is heavy on requirements, you'll probably benefit from more up-front exploration there, maybe just learning the biz domain -- you might need a deep grounding before you could successfully iterate on architecture. If it's architecture-heavy, you might be doing a lot of up-front architecture prototypes, walking skeletons etc, to dial down the architectural risk -- again, you'd want to do that before beginning to iterate on code. On the other hand if you know the domain well already and there are no significant unexplored architectural issues, you might want to head right into a Scrum. So the depth of the project in various areas would probably influence your choice of how much, and what kind of, up-front work to do.

 

-- Steve