OK, so a friend (who shall remain anonymous) sent me a message with some very good points bout estimation. This blog post explores those statements. It will be long, and detailed. Grab a cup of coffee, sit down and let’s do this!
The first thing I’d like to comment on is the statement
My start point is it is always possible to estimate accurately
— an anonymous friend
Let’s unpack this statement in detail:
Evaluating “It’s always possible to estimate accurately”
a) “it’s always possible to estimate accurately” There are 2 ways to look at this.
The first is theoretical: Given that we have data points (speculation about implementation choices in SW), can we generate a model (e.g. based on expert opinion) that can reliably (how often? 90% of the time?) deliver an estimate that is within an acceptable range of the actual (what’s this? 10%, 20%)
The second has a more practical nature: what decisions can I make, or do I need to make based on estimates, and what kind of accuracy do I need to make those decisions “viable” or “survivable” (survivable means: I can survive a BIG mistake in the decision).
The theoretical take on the problem of estimation
On the “theoretical” problem of estimation: this problem seems possible to resolve at face value.
However, the sources of the data (speculation by experts) must be within an acceptable deviation of the choices actually made by developers. This is not always, (perhaps even not at all) sure. Example: the expert assumes an API is usable which, upon inspection, reveals itself to be unusable for the desired functionality, therefore leading to a re-implementation of an API or other changes not accounted for in the original estimate.
If the data points generated by expert speculation are wrong, that makes the source of data points unreliable and in turn that makes the basis for estimation wrong. If this is true, all the other factors in the accuracy of estimates are inconsequential, as they depend on the accuracy of the data points.
This creates a big problem for estimation as a practice. In estimation, the data that is generated to create the estimate is itself in question. It may be totally off!
The practical take on the problem of estimation
When it comes to the practice of developing software, however, the problem is much different. It is not about “accuracy” of estimates, but it is about the survivability of the project. In other words: can I make decisions that, when wrong (and some will be), don’t “kill” the project?
When we define the goal of a decision to be “increase the chance of the project surviving”, then estimation is not a critical factor. Other factors are more important, for example: risks of catastrophic failures in software.
There is, in the Agile community, a shared understanding that the best way to reduce of a catastrophic failure is to release smaller changes, and release often (principle #1 of the Agile Manifesto!)
When we release software every day (I’m just now recording a series of episodes on Continuos Delivery for the Scrum Master Toolbox Podcast), the risk equation changes drastically. Given that the largest risk for a software project is that it delivers “nothing” (SW that cannot be used, for whatever reason), we must always prove to ourselves and the stakeholders that we have something that delivers value (can be used for the intended purpose).
By delivering software every day (for example), we are directly tackling the biggest risk to the project. However, if we deliver every day, then the actual delivery will impact what is “the right thing” to do next. This process of adaptability through collected feedback is the core of Agile.
It follows from this that, when we deliver every day, we effectively change the plan. That change increases our chances of delivering a successful project (our chances of survivability) and does not require us to be accurate in our estimates or even the expert speculation about implementation choices (we get feedback on small increments, and we can quickly adapt).
In other words: When we deliver every day and collect feedback from customers/stakeholders, we will change the plan to the point of making estimation obsolete, yet still increasing the chances of survival for our project.
That quote above carries on in a way I think material. There are a bunch of other points too.
The quote continues
“My start point is it is always possible to estimate accurately but that may cause so much imprecision as to be useless for decision making under uncertainty AND it requires sophistication/ skill/ maturity by both the creator and consumer of the estimate to share understanding and often either or both lack the ability.
So it seems to me there are cases where a better dialogue between investor and contractor (developer) may be possible by approaches that avoid classic estimating.”
The post above seems to imply the estimator and the do-er are different which may or may not be the case and it seems to suggest an estimate is static – IMHO an estimate is a machine or algorithm into which we put data and out of which we get a range of values. As soon as we find the machine or the data can be improved we do so.
Also IMHO it is the duty of the creator to ensure accuracy by using appropriate ranges. It is the duty of the investor to assess the range (uncertainty) and if its not suited to their risk tolerances to pay in time/ effort/ research/ spikes etc for improvement of the estimate – which generally means increasing precision while maintaining accuracy – accuracy is binary.
In part the choice of development lifecycle is driven by the nature of the work – is it Runner-Repeater-Stranger-Alien? If R or R estimates should be accurate at highish precision and a plan and design first approach is feasible (and best). If the problem is Stranger/ Alien or requirements are volatile the lifecycle should be short, iterative, delivery incremental and accurate estimates start with wide ranges. We may also choose to move the dependant-variable from cost/time and fix these so now scope (and hence ROI) become the dependant variable.
Releasing small and often does not avoid catastrophic failure! That’s 100% specious! A single bit can be the pivot of catastrophic eg light != Red versus Light = Red and simple stats says the more releases the more chances to screw-up a stable environment. I do agree the frequent automated delivery increases our fluency in releasing and so reduces one class of risk – incorrect deployment – and may via feedback help a customer develop their requirements – but it does not reduce many other risk types (which I assume you’ve introduced as the reason the estimating process can get broken. If we have a 1,000 elements in the project and we estimate them all then probability allows analysis and when we get screw-up analysis is also possible.
Practices for projects that are Runners (Done many times) vs Repeaters (Done a few times) vs Strangers (Done by others but not be us) vs Alien (Never, so far done by anyone) projects should be different