How do you prove that estimation is a scam? Well, it’s not easy, especially if luminaries in the Agile community defend that you can honestly estimate a project where you don’t have a team available and only a fuzzy understanding of Requirements or Technology involved, or both!
The fact is that many projects are indeed estimated before a team is even set up, but that is ok because we are armed with “expert” estimates! So I decided to test that particular aspect of the estimation “game”: can experts estimate better than non-experts a software project?
To be able to answer this question I have to design an experiment. It should avoid bias, and focus only on the estimation process and it’s outcome.
It should focus on the usefulness of an estimate for a particular project go/no go decision: accuracy of estimation. The parameters known should be a description of the problem (the problem has to be relatively common, or estimation would not be possible anyway) and the technology choices made for that particular project. And the resulting actual length of the project must be independent of the estimates.
The experiment
So here’s my first try at defining the experiment:
- Collect information on a project that has already ended (i.e. the length can no longer be affected)
- Ask 10 people with expertise in that area (problem and technology) to estimate the length of the project given a team size
- Ask 10 people without expertise in that area (problem and/or technology) but with expertise in software development in general to estimate the length of the project given a team size
Here’s my expectation (or Hypothesis 1): I believe that the accuracy of estimation will not correlate with the expertise in the technology and problem domains. If I am correct, this will invalidate the concept that expert estimates are somehow “better” (more accurate in the context of this experiment). The obvious conclusion should then be that the accuracy of estimates in this type of context (very common when bidding for a project) is at least significantly affected by other variables rather than expertise. If I am not correct, and the experts can indeed deliver reliably better estimates than the non-experts this should mean that expertise is an advantage when bidding in this type of projects.
Hypothesis 2 focuses on the usefulness of this type of estimates, i.e. their accuracy and therefore relevance for go/no go decisions. In this case I don’t have a clear bad/good line between different deviations, I’m just exploring the actual deviation and hopefully will run other experiments later on to explore the impact/meaningfulness of these deviations.
Now I need your help
I need you to help me design/improve this experiment, so that we can continue to further the investigation into Estimates and its alternative #NoEstimates. Help me out by answering these questions:
- Can you identify in the experiment description bias for or against expert estimation?
- Do you have other Hypothesis that we could test with a similar experiment design?
Let me know what you think about this experiment. Once the experiment has been reviewed by enough people and we have improved its design enough I’ll start preparing the data collection. For now help me fine-tune this experiment.
Photo credit: CC BY NC ND by Alvin K @ flickr