Software Best Practices

Okay, so I ran into a problem today.

What in the heck do we really mean when we give an effort estimate and say...

"I think best case this will take me 4 days, worst case is 16 days, and the expected is 6 days."

Is that "expected" we talk about the mean, the median, the mode, the expected value?

Let me back up... we say that estimates are statements of probability.  Let's assume for the sake of argument that our estimates have a confidence of 68%.  So in the example above there is a 68% chance that the actual effort will be between 4 and 16 days.  Now there are a buncha different ways that I can draw a probability curve so I come up with 68% of it being between 4 and 16.  Assuming a Normal Distribution (yeah, yeah, I should be using some other distribution, but the math is easier with the Normal) I get plots like this...

 And while that's all hunky-dory, the thing is that in order to fit it I needed to change the one free variable that I had, the standard deviation, from 4.1 if expected = 6, to 6 if expected = 10.

The PERT methodology uses (usually) the Beta distribution and in PERT they say "most likely" rather than "expected". In that case we are looking at the statistical mode if I'm not mistaken.

So what do we mean?

And why am I asking?

Well, I'm trying to figure out how to sum up the probabilities of completing a long chain of tasks to come up with the probable distribution of ed dates for a project.  But in order to do that I need to know what the distributions are for the individual tasks.  And to know the distributions for the end dates of the individual tasks I have only the effort estimates given to me by the team.  I can ask the team to estimate anything I want as long as it come back as low estimate, high estimate, and expected.

In order to turn that into a probability distribution I need to answer, "So that expected... is it the mean, median, mode, or expectation value?"

How do you folks handle this?

I've noticed that I can't find any tools that handle estimating in ranges in a graceful way.  Do you folks know of any?

Well I don't do that much planning but I tend to think of three point estimating as best, most likely, worst.
The most likely means that I'll come across a few problems along the way but nothing major that will throw wthe hole estimate out. The most likely is never the mean.

 For example I might say that a given task will take, 3,5,10 days. I.e. I know that the fastest I could do it in is 3 days if we didn't hit any trouble. More likely it will take 5 days.

Now if that task was for something I planned to do in two weeks time I would have a high confidence in it. If it was something 3 months down the line then I would have a far lesser degree of confidence in it.
 

Now think of your plan as a cone with the point as the start date. One dimension of growth is estimation. The other is confidence.

Now consider what happens if you apply iterative development with regular feature checkpoints? You can reduce or set back your  X and Y dimensions, effectively turning your cone into a straight line to represent what has been delivered connected to a cone for what is left to be delivered.

 So the best advice I can give is never plan too far ahead and make sure you use rapid development where appropriate!
 

Normally "expected" = expected value, i.e., the mean.

With PERT, you take (1*low estimate + 4*most likely estimate + 1* high estimate)/6 and that equals the expected value, i.e., again the mean. You have to supply the most likely (nominally, the mode).

brucehenry:

I'm trying to figure out how to sum up the probabilities of completing a long chain of tasks to come up with the probable distribution of ed dates for a project.  But in order to do that I need to know what the distributions are for the individual tasks. 

Actually you don't. The Central Limit Theorem says that summing *any* random variable will give you a normal distribution. So it doesn't matter if the individual tasks are beta distributed, poisson distributed, etc.  The sum will be normally distributed.

The approach I like the best is to do individual task-level estimates using PERT (high, low, and most likely), and then combine those to get the aggregate estimate. I discuss that in Chapter 10 of Software Estimation: Demystifying the Black Art.

The big warning here is watch out for Garbage In = Garbage Out. There's no point in worrying about whether the distributions are beta or normal if the means of the individual task-level estimates are off by a factor of 2. The key here is working with your team to set up a feedback loop so that they will get their task-level means as accurate as they can. If you can accomplish that, then the other questions are really second-order questions. If you can't get their means to be accurate, then the other questions won't even matter.

I don't have first-hand knowledge of any tool support for entering estimates in ranges. My understanding is that the current version of MS Project allows tasks to be entered in ranges and also provides support for Critical Chain scheduling, but I'm just repeating what I've heard other people say and don't know that from personal usage of Project.

 Good luck!

DavidH:

The most likely means that I'll come across a few problems along the way but nothing major that will throw the whole estimate out. The most likely is never the mean. 

David is partially correct on this.  The range between the best and worst case estimate communicates the risk associated with the estimate and the most likely case communicates the confidence in the estimate. It can be anywhere between the best and worst case, including (by coincidence) the mean.

For example, after looking at the requirements, my design lead decides to reuse a module. So she says her best case estimate is  x staff hours assuming that the module just needs to be integrated in and tested.  She also gives the worst case estimate of y staff hours assumes the module will not work and she will have to write it scratch.   Her most likely case can be anywhere in between.  Let's say that she is fairly confident that the module will work -- it's been used in several similiar applications. So she says her most likely is estimate is closer to the best case estimate.  On the other hand, let's say that she really isn't sure -- It's been used before by in a very different application.  In that case, her most likely will be closer to the worst case estimate.  Then you can use the PERT formula to compute the expected case.

Cool.  This all matches up pretty well with what I was thinking.

Reading Steve's reply I realize that mathematically what PERT is asking for is in fact the mode (most likely) and that the expected value needs to be calculated from that.  I think that my confusion came from the fact that for most reasonable estimates the expected value and the most likely value are very close together.

Jerry points out something that I should have realized which is that the most likely value can quite by accident be the mean.

One thing that I forgot to mention about the two curves that I included in the original post is that both are 80% confidence ranges.  That is, I have the same confidence in each, just for the red task the most likely effort is 6.  I realize now that I mislabeled that legend on the graph.  It should have said "Most Likely" rather than "Expected".

Thanks for the help!

Bruce P. Henry

LiquidPlanner

Another approach to getting an aggregate probability distribution for the end date of a schedule, based on range-based estimates for individual tasks, is to apply a Monte Carlo simulation to the problem. I have done so very successfully using a proprietary tool that we developed in-house, but there are commercial tools out there that conceptually do the same thing (e.g. Risk+, which plugs into Microsoft Project). I hope this helps.