Blink Estimation

Experienced delivery folks can have surprisingly good instincts for macro-level estimation, as long as we are careful to manage blind spots and cognitive biases. This can be an important tool in early project investment discussions, and can remove roadblocks where people are uncomfortable or unwilling to provide estimates.

Prologue

Back in 2003 I was working at an Internet bank with the remarkable Dave Leigh-Fellows. I could tell you many stories about Dave. One of my favourites involves roller skates. On this occasion we were running a pre-planning session for a major new product that would integrate all your bank accounts into one place. You could see at a glance your mortgage, savings, loans, current accounts, credit cards, everything in one place, from any number of banks or financial institutes. It was quite an audacious plan, and understandably there were many risks and integration points. Before we even got into release planning it was decided we would spend a day thinking about how that planning should work. It was like the United Nations, with 18 people sitting around a large, U-shaped table, representing technology, the business, senior management, operations, legal and compliance, and no doubt various other folks.

We were about two hours into the day and things were moving like treacle. The sticking point was scope and project size. How big was the box? How many people and how much time would it all take? We kept going round the same meta-discussions: What could we do to start to discover how much effort would be involved? Exhausted and frustrated, we stopped for a break. In the hallway Dave said something to me. He was convinced all the people in the room already knew how big the problem was, but no-one wanted to go out on a limb and suggest the scope for fear of getting shot down. So he came up with an idea. As the external consultant I was to be the fall guy. I would suggest Dave’s plan as though it were my own — no-one would have listened to Dave because he was “just” a delivery lead and there was too much at stake. Once we were all sat back down I suggested an exercise. We had been talking for over two hours now and everyone had a sense of the project we were taking on and the parameters involved. We had confidence in a third party vendor who would supply the links into the various external banking platforms, so the real product would be the application on top of all this to wow our customers.

I asked everyone to take an index card and to write on it, based on nothing but their gut instinct and what they had heard so far, how many people and how much time they thought the project would involve. I suspect the number of raised eyebrows outnumbered the people in the room, but I said that the whole exercise would only take five minutes and asked them to humour me. You can imagine what happened. On a count of 3 we all raised our cards, and everyone’s estimate was the same, around 6-8 people for around 6-8 months. Some were towards the higher end and some lower, but no-one was out by an order of magnitude and everyone had an opinion.

“Well,” said Dave, laughing, “it looks like we’ve got our project estimate. What’s next?” Indeed we had a consensus around the room that 8 people for 8 months should be able to take a reasonable-sized chunk out of the problem. Eight months later, a team of eight people delivered the first release of Money Manager and it was a great product.

The second time

I remember being completely blown away by how well this exercise had worked. And then promptly forgot about it for five years. The intervening years were filled with ever more detailed and sophisticated planning techniques. The zenith of these was a multi-column spreadsheet detailing, story by story, minimum, likely and maximum effort, numbers for clarity and volatility, and other things I didn’t really understand. I’ve written about this before, but I didn’t tell you what happened next.

I came out of one such planning exercise — two days of gruelling micro-planning and estimation that had generated in excess of 400 stories in amongst some genuinely useful conversations — and was greeted by one of my colleagues who asked why I was looking so ragged. I told him we’d been in a two day planning workshop for a business website. He asked what our final rolled-up estimate was for the project so I told him.

“What??” he exploded. “My grandmother could do it for that much!” And sadly he was right. This was the shot in the arm I needed to realise that a) micro-planning is fractal: The more detail you go into the more the rolled-up result expands, and that b) there must be a better way. The fact that everyone else in the agile world was doing it this way didn’t make me feel any better about it.

A couple of days later we got an inbound call from a TV company looking to rebuild their web platform ahead of a new season of a reality show. The site traffic would be very spiky: after each episode there would be a two hour window during which several million people would register a vote for the person to be ejected from the show. The rest of the time it would be just a regular media site with forums, news updates and suchlike. I decided to try an experiment, and as I look back I realise it must have been informed by my earlier Leigh-Fellows experience. I took a small number of people into a room. The group included a couple of senior developers, a tester, a business analyst and a project manager. I seem to recall the analyst had some previous media website experience. I described the project brief to them and we discussed it for a while, maybe 20 minutes. What kind of traffic could we expect during the voting window? What sort of resilience and redundancy should we plan for? What about tying in with their media style guide and design constraints? What kinds of technologies were at our disposal? How much access would we have to the client, and how frequently?

Then I did the index card thing again: I asked each of them to write down roughly how long it would take and how big a team they would need. Extra credit if they had specific skills or people in mind for the team. Again there was a surprising degree of consensus, so I called the client and gave him the project estimate.

“How did you come up with that number?” he asked.

I could have bluffed it but instead I said “I got a group of really smart people in a room, with at least 10 years’ experience each, and asked them.”

After a pause he replied: “Sounds like a pretty sensible approach to me. You’ve got the gig.”

Sometimes you just don’t know

Of course now I had a problem. I had stumbled upon a fast, accurate way to estimate at project scale, and I was pretty confident no-one would believe me. It simply didn’t have enough voodoo. I tried it a couple more times and each time it seemed to just work. I wasn’t allowed to run free with this of course. The pseudo-scientific micro-planning would still take place in parallel. But my blink estimates, as I had taken to calling them, were at least acting as a foil to project inflation. “It just doesn’t feel like a one year project” was often enough to push back on some of the more bonkers estimates.

Then I hit a stumbling block. A client had asked us to estimate a piece of work and I had pulled a group of people into a room. They knew the drill by now and we had the discussion and did the estimation piece, and we all held up our cards. They all had a number of people and a number of months, except one card that the business analyst was holding up. Hers just had a large question mark written on it.

“What does that mean?” I asked. “It means we can’t possibly know, and here’s why.” And she proceeded to list half a dozen reasons why we couldn’t possibly know. As she was speaking, we all lowered our cards, feeling increasingly silly. She had driven a truck through any possible blink estimate. So I went back to the client and told him. We can’t give you an estimate but we can describe where we see the uncertainty, and we suggest you spend a couple of weeks investigating some of these unknowns, in order to come up with some kind of sizing.

The client was quiet for a minute or two. Then he said: “That’s interesting, and I believe you. So now I’m left wondering how the other suppliers I asked were able to come up with an estimate. They must have been just guessing, because surely they have the same holes as the rest of us.” On the back of this he gave us the investigative piece, and we went on to win the follow-up work once we had given him an idea of the size of the main delivery.

The rules of Blink Estimation

I’ve been teaching Blink Estimation in my Accelerated Agile class and introducing it in my consulting work for about a year now and a number of people have been trying it. I usually recommend running it in parallel: do it alongside whatever you currently do, and if it differs significantly keep with your current method but maybe question its veracity. The results have been surprisingly positive. One of my clients told me they used blink estimation to break a planning deadlock where for various reasons people were uncomfortable about giving estimates he knew they had in their head.

There are a small number of “rules” to increase the likelihood of it working, and a number of caveats. First the rules. There are three things you need to do blink estimation:

  1. Experts estimating
  2. Expert messenger
  3. Expert customer

Experts estimating

This should be obvious. Blink estimation is a comparison exercise that draws on the context you’ve built up becoming an expert, in the Dreyfus sense. It says: Based on all the projects you’ve delivered in the past, and given what you know of the current context and constraints, how big do you think this new one is? Without the context of many other projects you simply don’t have a suitable frame of reference for estimating this one, so it requires the experience of many previous projects, both successful and unsuccessful!

Your role as facilitator is to gather people from different disciplines: testers, analysts, project managers, programmers, architects, operations and support engineers among others. Choose disciplines that are likely to have overlapping but distinct areas of expertise. You frame the discussion using neutral language and open questions, and try to dissuade the participants from over-thinking things. Using words like roughly or about frees the participants from feeling committed or constrained by their estimates. And remember, these are just estimates.

You want to minimise the likelihood of shared blind spots, which is why you want people from different backgrounds discussing the potential risks and pitfalls. You’ll be able to sense when people are moving towards a decision point. At that point you ask everyone to estimate size and duration, and then if there are large variances you can explore the assumptions behind the outliers.

This is just the same as planning poker but at project scale. The security expert might see an online store project as much harder than a developer, because she knows the enormous variability that PCI audits, injection attack testing and other hardening activities can introduce. The developer can then put her mind at rest by suggesting they use a third party pass-through credit card processor to remove PCI risk, and asking her to be available at each mini planning session to advise on where the security dragons might emerge. In practice I’ve not seen macro-level estimation take more than a couple of rounds, with diverse conversations and discussions at each round, to reach a reasonable consensus. It’s worth repeating, this is an estimation session. We want an order of magnitude, not a precise answer.

Expert messenger

The expert messenger is about knowing how to frame the output of the exercise so it makes sense to the customer. You need to be comfortable describing how experts rely on instinct and intuition, and defending that as a legitimate basis for estimation. You should consider the various cognitive biases I describe below, such as anchoring, priming and environment, that can impair the estimate. One counter-intuitive outcome of blink estimation is that it doesn’t require the delivery team to estimate their own work. As long as the experts are aware of who will likely be delivering, especially if this materially affects their estimate, and as long as the team trusts the people doing the estimating, then it still works.

Expert customer

Having an expert customer is the condition least likely to be under your control. I was lucky the first time, having Dave in the room, and the second time with the media client who just “got it”. Back then I doubt I would have been able to convince him. If he’d demanded the full story breakdown I would probably have just caved and spent two days building the spreadsheet. Nowadays I recommend investing effort into educating your customer or client. Once they are on board with the idea of trusting the delivery team’s experience lots of things become a lot smoother. I’ve even done blink estimation sessions with the customer as part of the group. It’s pretty powerful!

Blink estimation as a learning workshop

So where do these expert estimators come from, and how do we grow the next generation of them? Once a team is used to the dynamics of blink estimation you can use it as a learning workshop for the non-experts. The non-expert observers are just that: observers. They are not allowed to ask questions in the blink estimation session because their questions can unintentionally lead to priming or anchoring biases, but they can take whatever notes they like and afterwards they can ask as many questions as they want. One that often comes up is “Why didn’t they discuss X?” Where X is someone’s pet topic, say security or testing. It usually transpires that everyone was aware of X, but no-one thought it would affect the estimate so they didn’t bother raising the subject, and this is the entire point of blink estimation. The estimators only discuss issues they feel could materially affect the estimate. Everything else, however important a topic in its own right, can be safely deferred until the project is under way.

Programmed to fail

The huge elephant in the room with blink estimation, and the reason I recommend trialling it alongside whatever you currently do, is that any instinctual exercise is subject to any number of cognitive biases. From the story you read in the newspaper on the way to work, to the temperature in the room, to the last number you heard — however irrelevant it might be — your instinctual self is subject to unexpected recalibration at a moment’s notice. Daniel Kahneman won a Nobel Prize for showing us how easily fooled we can be, and how utterly confident we are that it only happens to Other People. I highly recommend his book Thinking Fast and Slow. As a facilitator of blink estimation your job is to try to manage those biases. For example everyone showing their estimates at the same time is one countermeasure. If we went round one at a time the earlier estimates would act as an unconscious anchor to the later ones.

Estimation as an investment decision

When someone asks me for an estimate I try to invert the conversation to talk about investment. When they ask how long you think it will take, they are really asking: How much should I expect to invest in this in order to get a decent return from it? If you can surface this question instead you can help them explore the expected return, and it becomes a straightforward return-on-investment discussion rather than the traditional estimation-as-contract negotiation.

Once you have agreed an investment in terms of people and time and other resources, you can choose to deliver to that timescale. By attempting to surface uncertainty early on and keeping an eye on the goal of the project, you simply work towards the date and declare victory when you get there. In reality you can often declare mini-victories on the way through, with interim releases to demonstrate progress and validate your assumptions. It’s like firing an arrow and then painting the target around it — you start hitting a lot of bulls-eyes! Focusing on uncertainty early increases the likelihood of surfacing assumptions that can have a material impact — either good or bad — on your original project estimate. This allows you to provide your delivery assurance and governance functions with information to support effective steering.

So what’s wrong with story-level estimation?

Every project contains uncertainty. If it doesn’t you shouldn’t be doing it — it’s already been solved! At a macro level you can have an educated opinion about how much space you should allow for dragons, those unknown unknowns that are waiting to derail your project. With sufficient experience and the right mix of people in the room you can reasonably assess how this project is similar to, and different from, previous ones, and how much you should reasonably invest to realise the business impact associated with it.

Breaking things down to story level is one way of exploring a space, and can lead to some good discoveries, but it far from the best approach to surfacing uncertainty. If it were, all agile projects would contain no surprises because they would all have emerged during the backlog creation. Except that means we’ve just reinvented big, up-front analysis and design! We can’t know which individual stories will surface a particular dragon, not least because many surprises are cross-project rather than specific to a particular story, like unexpectedly unavailable hardware or unexpected organisational constraints, so it’s pretty random which story will be first to the line. At delivery time, one three-point story might take five times as long as another, for very good macro-level reasons. Where then is the value in estimating each story?

On the other hand, a group of people with complementary experience and skills, having a conversation whose sole intent is to identify gaps and surface assumptions, can quickly reach a strong consensus on how big the box really needs to be.

18 comments

  1. Hi Dan,
    Thanks for this post. I found it really interesting and very similar to our experiences here at On the Beach during our agile journey over the past few years. I blogged on a similar (although not as detailed and articulate as yours!) subject a while back: http://scrumrat.blogspot.co.uk/2012/09/why-we-no-longer-do-estimating.html. They’ll be a bunch of us at the Agile On the Beach Conference in Cornwall next month. It’ll be good to hear more of your experiences.

    Paul

  2. Dan,

    Thank you for this excellent article. I have actually been performing this sort of activity in my consulting over the years but never put a name to it. Also, I appreciate your insights and caveats. It is always refreshing to find someone putting words to something you do without much conscious thought because it just works and falls into the background of your every day activities. I look forward to reading more of your articles.

    Our company has just begun working on a new proposal and I intend to apply “Blink” along side our traditional estimation model.

    Paul Soutner
    Founder, Megaphone Studios

    1. Fantastic, let me know how it works out. If it is possible it would be great to have the blink estimators and regular folks separate and unaware of each others’ results until they are both done. That way there is less likelihood of cross-contamination. Of course if you have the same people doing both you’re going to have the results influencing one way or another.

  3. Hi Dan,

    Good article, thanks for sharing.

    Reflecting back on these projects you ran that were kicked off with a Blink Estimate, how much of the success in roughly hitting the date with roughly the number of people everyone expected was due to knowing the boundaries you had (time/people), and over the course of the project adjusting the scope accordingly?

    Doing good feature work, but only doing what was necessary in business priority order, is the quickest path to MVP. And once you have an MVP, everything else is just gravy.

    Thoughts?

    Thanks again.

    1. That’s what I mean about shooting the arrow and then painting the target around it. Because we all agreed what a suitable investment looked like, it made it easier to have scoping conversations. “Yes we could include that feature but would that blow our schedule? What’s really the priority here?”

  4. For precisely these reasons I’ve started out StarterSquad without doing estimations. It turns out that there is a lot of stretch in the amount of money you need to build an MVP. I’ve found that by doing ever more aggressive estimations, jobs would get completed faster with less people. The client satisfaction would stay constant. This was linear, until I hit the bottom.

    My theory is that virtually all software organisations are ridiculously overestimating, and then smart developers make sure to at least reach the minimal goals and eliminate enough waste to keep the customer happy. As you know, if there is room it will be filled ( as a dev I can argue that I need that new templating framework to do my job perfectly, and if the client is fine to pay for it I’ll happily build it and use it on the project).

    It follows that the _only_ thing impacting costs is budget in most cases. Once you lower the budget to a point that the math of developer productivity gets interesting again only then do your experiments produce valuable results.

    I’ve done this with StarterSquad and started estimating again. I’d love to hear your opinion on this detail.

    http://www.startersquad.com/blog/startersquad-pricing-model/

    1. Interesting. What I read from this is that by introducing an extreme budget constraint and effectively asking “What could I get for this tiny investment?”, and “How tiny does the investment need to be before the return isn’t worth it?”, you are learning that you can get value out of surprisingly small funding increments.

      I definitely think there’s something in this, but I suspect over time the minimum funding increment would need to grow as you decide to create more sophisticated capabilities. I wonder how much of this low-hanging fruit there is early on in product development? I imagine it could run for quite a while.

  5. Thanks for posting these stories. In my view they highlight that estimations are not (should not be?) complicated endeavours. In fact you state that several times you were able to give an estimate with a simple questions and a “blink” estimate from what you call experts (more on that later).
    At the very least this means that estimations do not get better in proportion to the time allocated to creating them. (Many would not agree with this).

    But later in the article you make assumptions / assertions that you do not support with evidence. Have you tried the same exercise with non-experts in the mix? (but the others should not know who they are)

    The first example you describe could be an example of estimating what is “acceptable” cost/time instead of a real estimation. Or it could be just an example of “wisdom of crowds”.

    Your post makes me ask many questions, it opens a path of exploration: why are blink-estimations even needed? I hope @jacoporomei will comment here as he has been doing contracting *without* estimates with success for both him and his customers.

    I also hope that the estimation supporters will comment on what they find when reading the stories you describe.

    1. Hi Vasco,

      You’re absolutely correct that this isn’t even close to a scientific study. I’m not sure how you would set up the kind of experiment you’re describing, with a “ringer” in the room, because you would expect everyone to know each other. It might be interesting to get several groups of different permutations to blink estimate and then comparing the results.

      As I’m reading more of Thinking Fast and Slow I’m seeing a number of indicators that can improve the likelihood of blink estimation working, such as getting the group to think about previous projects of similar size, complexity, uncertainty and various other dimensions, to promote more contextual “outside thinking”. Of course I’m also trusting my own instincts less and probably subject to confirmation bias so I expect it balances out!

      To answer your question, some kind of estimate is needed to support an investment decision. “Give us some money and we’ll see how far we get” doesn’t usually cut it with the budget holders. They will quite reasonably want a sense of the likely cost of various options so they can make good governance decisions. Of course the long game is to move towards throughput accounting and beyond budgeting, but most large organisations are a long way from that yet. Blink estimation seems to be a reasonable middle ground for providing “accurate enough” information at an acceptable cost.

      1. They way I have understood the Beyond Budgeting, they still rely on some kind of financial estimate before starting a project. However, what they do different is that they are much more liberal about what projects to start, and less obsessed with the initial estimate. Instead they spend their energy on watching the progress of the project closely, using e g activity accounting. If a project does not seem to deliver, they cut it short and early – much like the Lean Startup approach.
        I know the both of you (Vasco and Dan) agree with me that more software projects should be run like this. If you never cut a project short, you are not doing much of a portfolio management.

  6. […] See on Scoop.it – AgilitéExperienced delivery folks can have surprisingly good instincts for macro-level estimation, as long as we are careful to manage blind spots and cognitive biases. This can be an important tool in ea…See on dannorth.net […]

  7. Thanks Dan for sharing these stories with us: very interesting and – in a remarkable amount – meaning a bit of relief for my recent challenges in eBay Italia.

    Vasco – thanks for calling me in – is referring to my experience with super light contracts I have been using for 2 years which only featured these 4 simple rules:
    – Very, very, verrrrrry short iterations
    – Fixed price (as in ‘price’, not ‘cost’) for each iteration
    – The customer is let free to reject the iteration deliverable, paying nothing
    – The supplier can apply discounts for equity or revenue shares

    These simple 4 rules let many *good* epiphenomena emerge. Among these the most relevant here is that, all of a sudden, the customer is not micro-managing your estimates anymore. Even better: they don’t *care* about your estimates at all!
    This lets the (whole) team free to “blink estimate” – if they want – but above all to focus on next delivery, trying to deliver value enough here & now to convince the customer that
    – their work is worth the money waiting for them by the iteration end
    – the direction followed by the project is the right one iteration by iteration

    1. I like the sound of that approach. What is the largest size project or product you have been doing this with? I’m wondering whether it would scale to delivering a portfolio of interdependent workstreams.

  8. Hi Dan,

    Very interesting article, made me think of similarities in the way an experienced team who are well-knit are able to deal with inconsistent / drifting specs.

    I suspect the Expert customer is more often the lynch pin in getting acceptance for the technique. Thanks for documenting and posting.

  9. […] Don’t break down every story in the beginning and add it up to find out how long the project is going to take, this will just leave you with an unmanageable backlog and most of the stories will be out of date by the time you come to develop them. We have found that the best approach is to get a group of experts in the room and play planning poker for the project as Dan North suggests calling it Blink Estimation. […]

  10. […] Blink Estimation by Dan North Another take on a planning poker variation. […]

Follow

Get every new post delivered to your Inbox.

Join 476 other followers

%d bloggers like this: