DanNorth.net

I’ve been having difficulty recently with a trend I’m seeing on projects. What happens is this: we (an agile development team) engage with a client for a short period – typically 2-4 weeks – to investigate how we might work with them to solve their problem. During this period, we drive out lots of stories, do some technical investigation and estimation, and out of this we generate a high level release plan. All very good, you might think. But then, unfortunately, everything is mysteriously forgotten about except the release plan, which suddenly grows horns and fangs and becomes a Fixed-Price Fixed-Scope Big Up-Front Project Plan. Well ok, it’s not quite that dramatic, but it certainly doesn’t feel very agile to me. Hopefully I can explain why.

To my mind, there are two primary objectives of this initial phase. Firstly, we get to know each other and think about how we want to work together. Secondly, we create a shared understanding of the project’s objectives, timescales and risk profile. Coming out of this process, everyone should have the same sense of “bigness” of the project, in terms of both size and complexity. The release plan is just a detail in the process.

An agile project – in fact any project – should focus on a set of outcomes. How we get there is less important than actually getting there. A CIO or business sponsor wants to solve a specific problem, by a specific date, and has an idea of how much money they are prepared to spend achieving that. By driving out a comprehensive list of stories, estimating them, and then rolling it all up into a Big Release Plan, you run the risk of focusing attention on the features (stories) and defining Success as the delivery of all these stories. This is exacerbated when the release plan is shown to people who weren’t part of the original exercise. They see the release plan and naturally assume that it is the primary “output” of the process.

In the manner of the Agile Manifesto, whilst there is value in delivering a list of features, I value achieving the outcomes of the project higher. This feels complementary to the other four agile manifesto values.

Two recent examples. On one project it became apparent that we were unlikely to meet a particular hard deadline given the current story backlog. The team worked with the client to find a way that they could still meet the project’s outcomes and deadline, by dropping or deferring around a third of the features. The client was delighted! The message was simple: I can meet your outcomes with only two thirds of the effort (i.e. cost) of the previous plan. Who wouldn’t want that? That conversation would have been dramatically different if the client had been wedded to the feature list as the definition of success of the project. Luckily he had the insight to remain focused on the overall objectives of his project, and they hit the date.

Another project had a stability issue in production. The (very critical) application would fail mysteriously every couple of days. They needed a solution now because they were losing business as well as being in danger of getting in regulatory trouble. The project manager simply asked the operations guys to manually restart the application every night as a standard operating procedure. At a stroke the initial problem was solved – the outcome was met – without a single line of code being written. Now the team could focus on solving the root cause, without being under pressure to produce a rush job.

“Scope” is a dangerous word, since it can be used to mean either features or outcomes, but never both. Does “cutting scope” really mean failing to meet objectives? Or were those features not really core after all? A project has to have a goal, otherwise you can’t know when you’re done. Without a goal, you will tend to define “done” in terms of execution of the story list. Similarly, it has to have a feature list, or you don’t know what you’re going to do to get there. So it is just as big a risk to have no clearly defined outcome (i.e. no vision or purpose for a project) as defining a fine-grained feature list up front. Next time you engage on a new project, make sure the entire team – and especially the sponsor – has a clearly articulated outcome for the project. And make sure you regularly review your direction to ensure you are still working towards those outcomes.

At the beginning of this year I wrote a feature article for Better Software magazine, which was published as “Behavior Modification” back in March.

The article is now available on my site. It gives an overview of behaviour-driven development, from its origins as a coaching aid for TDD through to its current form as a proven, comprehensive development approach.

When Darwin’s “On the Origin of Species” was published, evolution was already a well-established idea. Darwin’s insight was that evolution – through the process of natural selection – led to the introduction and extinction of entire species. This of course upset the established church, and to this day there are still religious groups who reject the idea, in the face of overwhelming scientific evidence.

The early twentieth century saw the rise of “neo-Darwinism”, which finally provided a mechanism for natural selection. It wasn’t happening at the level of creatures but at the microscopic level of genes. Natural selection is simply the playing out of a long-running stastistical game based on shuffling genes, and more disturbingly the creatures themselves (and ourselves) are just a very sophisticated form of temporary accomodation for those genes!

Richard Dawkins, the acclaimed scientist and author of “The Selfish Gene”, goes a stage further. He suggests that a species’ ability to adapt or evolve may itself change over time. In other words, evolvability is a trait that can evolve, just like a tail or gills.

Agility is another word for Evolvability

So what does this have to do with the world of software development? Well the various flavours of agile development, such as XP, Crystal, Scrum, Lean and DSDM, are all trying to solve the same problem, namely coming up with a repeatable, predictable way to adapt to change. In other words, they are trying to make software delivery evolvable. This ensures that delivery isn’t wrong-footed by a change in the project’s ecosystem or environment.

As an agile process coach, my job is to increase a team’s – or in the case of an Organisational Transformation programme an entire organisation’s – ability to evolve. Taking a cue from Mother Nature, you can’t expect this to occur overnight. The weight of evidence shows that macro-mutations in the process are almost always going to be detrimental. Instead you need to evolve the organisation towards evolvability, or agility, through a series of small steps that are easy and intuitive to grasp.

You can’t just walk in and announce that “this is how we do things now”. You need to take the time to understand and appreciate the existing ecosystem, and present the change process as a simple and natural way to improve delivery. At a getting-things-done level this is usually pretty straightforward: tested code tends to break less than untested code, and paired code tends to be better designed than solo code, so the incremental changes are intuitively a Good Thing.

At a project management or governance level the messages are more subtle. It’s not about how quickly or accurately you can deliver – you can put a bunch of bright people in a room with a big enough incentive (be it carrot or stick) and software will come out. Instead it’s about how resilient the process is to some sort of disruption.

Delivery Assurance addresses exactly this, and agile delivery is the most effective way I know of managing this risk. By pair programming you reduce the impact of a single person becoming unavailable or leaving the team. By applying the principles of test-driven (or behaviour-driven) development, you eliminate the risk of introducing regression defects into the codebase. Adaptive planning and tracking – and their resultant incremental steering and project funding – leverage this to provide programme-level adaptability. The requirements can change, the political environment can change, the goalposts can move, and you are still in with a chance of delivering something useful.

The more you can evolve an organisation’s evolvability, the more resilient you make it to change, and the better it can adapt to changes, both internally and in the external market. As Kent Beck famously said, embrace change. It is inevitable, and your chances of survival are directly linked to your ability to evolve.

ps. Thanks are due to Mats Helander for the casual comment that inspired this article.

Behaviour-driven development started life as an NLP exercise to stem the abuse of the word “test” in “test-driven development”. Since then it has grown into a respectable and proven agile methodology (with a small “m” of course).

Dave Astels, the award-winning author, was an early adopter of BDD and has been instrumental in raising its profile. He presented it at Canada on Rails and is taking it to JAOO and SD Best Practices. He has even presented it to Google. His Ruby BDD framework, rspec, has inspired a number of similar projects.

Now, Dave is a programmer. What’s more, he is a very thoughtful programmer, which means he invests a lot of energy in making programming productive and effective – and more importantly fun – for himself and other programmers. He doesn’t get very excited about capturing requirements or the dynamic between testers and analysts, so you won’t hear him talking about the wider context of BDD, but you will hear him saying that there is one!

BDD is fundamentally about identifying behaviour. At the analysis level, the behaviour of a story is its acceptance criteria, which BDD expresses in the form of automated scenarios. You need analysts working with testers to capture the stories and identify the acceptance criteria, and then you need programmers working with testers to automate the scenarios.

So the ironic twist is that all that talk about “testing” in TDD was taking the focus off the real testing action! My efforts to put the word “test” back in its box have in fact propelled the testers into the central role on a BDD team.

So the message here is twofold. Firstly, although a lot of the existing BDD message is in the TDD space, don’t lose sight of the bigger picture or you’ll miss out on all the good stuff. And secondly don’t underestimate the value of the testers on your team: they are your direct line to delivering high value software.

This question came up at the recent Expo-C conference, when presenter Michael Stal talked about designing services to be “as simple as possible but no simpler”. Kent Beck advises us to do “the simplest thing that could possibly work”, but this is often mistaken for “the first thing I could possibly think of” or even “the only thing I know” (also known as the Golden Hammer antipattern).

We picked up the topic as an open space session later that day. I took the position that you should only ever focus on the specific case in hand, and then only generalize if you meet another similar case. Erik Meijer, the reknowned language guru and thoroughly nice guy, took the opposing stance that you should abstract the general solution and then implement it locally. He pointed out that this was an approach with centuries of mathematical tradition behind it and is a fundamental tenet of modern functional programming¹. This developed into a good-natured but heated stand-off. I just couldn’t see the value of building a fancy general solution on the client’s money, when there was lots more important work to do and limited time. He couldn’t understand why you wouldn’t take the effort to make something as flexible as possible for the end user, especially if you were designing something like a programmers’ toolset or language. On the scale of specific to general – or concrete to abstract – we were at opposite ends, with a seemingly intractable chasm between us. He was simply wrong, and so, of course, was I.

If you are working on an agile project, you can confidently take the “simplistic” approach of only solving for today, safe in the knowledge that you can easily change the code tomorrow, or next week, as you learn more. If you are designing something that involves publishing an API – of which designing a programming language can be considered an extreme example – then your decisions can have a far-reaching impact².

This led us to the notion of a Change Event Horizon, which is the amount of time before you can undo the effect of a decision. For a programming language the change event horizon is huge. It can take years to repair a bogus language decision, and involve complex processes such as deprecation. For an internally-deployed enterprise app you are only as far away as your next release date.

So now we had a working definition of “too simple”. It can be defined as any decision that doesn’t consider the scale of the change event horizon. As an agile developer, one of my objectives is to create as small a change event horizon as I can, through the medium of continuous integration, automated testing and regular user feedback. Erik doesn’t have that luxury – he has to predict what his users will want, and use much blunter instruments such as beta programmes and focus groups. Add to that the myriad unpredictable things people do once you release a language – who would have predicted that Ruby would become the DSL host language of choice, or that VB would ever become cool³ – and you can quickly see that Erik lives in a far more complex world than me. No wonder then that he looks to the general, abstract case whenever he encounters a new situation. And I can rest easy knowing that my tiny change event horizon means I can change the world with the next release.

NLP talks about “in time” vs “through time”. Being in time means living in the moment, and not worrying about the future. Being through time means seeing the wider context, and carefully considering the implications of any action. Of course most people are a healthy mix of both of these. I am notoriously in time, which might explain why the immediacy of feedback on an agile project appeals to me – I simply don’t have the attention span for anything longer!

¹ Erik is also the father of Haskell

² Thanks are due to Niclas Nilsson for taking the role of translator/peacemaker for us to reach that realisation.

³ Take a look at some of the up-coming language features in VB9 if you don’t believe me.