At the end of 2009 I left the world of agile delivery and consulting to join a small team in a trading firm. I was member number three. The team grew to five in the next few weeks. This team was the most insanely effective delivery machine I’ve ever been a part of. A handful of programmers sitting in amongst a handful of traders, producing state-of-the-art trading systems – with all the integration pain (back office, risk management, connecting to electronic exchanges) that involves – in weeks. Not months, weeks. This is, of course, impossible.

They were doing it without following any of the agile practices that enable teams to perform, or “hyper-perform” if you believe the, um, hype. They weren’t doing TDD religiously. Some, but not religiously. They didn’t use continuous integration (!). They pretty much worked in silos – each person was responsible for one part of the stack, although the team had stand-ups so they all knew what was going on. They didn’t have a backlog. They certainly didn’t estimate stories or do release or even iteration planning. In fact they didn’t use iterations. And yet they were still delivering very high quality software, and their crazy pace was sustainable! This is, of course, impossible.

Coupled with all this, they had a management team – right up to the director of development – who got it. Which meant none of my super-powers worked. My ability to scale huge political hurdles with a single coffee was useless. I would push on heavy-looking organizational doors and they would simply fly open. There was a culture of trust and integrity that I hadn’t seen before. They even had an explicitly-stated policy of “no assholes.” And this was in a company of over 400 people operating in several timezones. This is, of course, impossible.

Getting my bearings

I went into what I can only describe as culture shock. I needed to either figure out what was going on here or just click my heels and head back to Kansas, where I at least understood the rules. So I decided to start observing and trying to articulate what was going on. That decision was three years ago. Since then I’ve been talking about this phenomenon using various names to describe it. Naming things is hard. To start with I called it Patterns of Effective Delivery, which required me to define the words delivery, effective and patterns. Surprisingly tricky, especially delivery.

Then I simplified it, calling it “Faster Software Delivery.” That didn’t last long, especially since I realized that one thing these guys weren’t delivering was software. They would solve complex business problems using tiny little apps and services – way smaller than the usual behemoths I was used to working with. No, software wasn’t the thing they were delivering – business capability was what they were delivering, using the smallest amount of software they could. “Faster Business Capability Delivery” doesn’t even sound right, so that wasn’t it.

Then I stepped back to look at what they were really doing. They were reducing the cycle time of delivering business capability from months to days. They were using agile values without being slaves to any of the big-A Agile practices. They really did value collaboration over contract negotiation. Their customers – the traders – were on the same side as them rather than across the table negotiating scope or delivery dates. They really did value working software over comprehensive documentation – although they would comprehensively document where it was necessary because they didn’t not value documentation. They really did value responding to change over following a plan. And they really did listen to one another and value each other’s opinions over any process or tool. Maybe this was what the Agile Manifesto was on about all along.

Sharing the love

So now I’m calling it Accelerated Agile. When I first started talking about Patterns of Effective Delivery in 2010, I had observed 11 patterns I wanted to share – by which I mean repeatable strategies that work in a given context and that resolve various forces while introducing others. Not a recipe or a guaranteed outcome – just a pattern. I now have nearly thirty, so it’s time to write a book.

In the meantime – and here’s the selly bit – I’ve gone independent and I’m running training courses in a number of places over the next few weeks and months. There are one, two and three day versions, going increasingly deeper into the material. More details are available on my site. I’m also going to be speaking at numerous conferences around the place.

The main reason I’m doing this is to learn. If you see me at a conference and any of the Accelerated Agile material resonates with you (“but we’ve been doing that for ages!”) then I’d love to hear from you. Where did it work? Where didn’t it work? I want the disasters as well as the success stories, otherwise it isn’t a pattern, it’s a sales gimmick.

I’m very excited about this – I really think it’s significant. The more I talk to people, the more it seems there’s a sea change occuring in the agile world. We’ve hit a local maximum with all the methodologies that came out of the 1990s. The context has changed, and many of us no longer have the huge, heavy organizational constraints of that decade. It’s time for a different kind of agile. It’s time to accelerate.

Moving on

Eighteen months ago I embarked on a three month experiment with CIO Seth Thomson to see if we could pair on a change programme across DRW IT, with me in London and him and his team in Chicago. In that time we’ve been applying ideas from Lean operations and Theory of Constraints to improve turnaround and communication across the firm’s IT group. We started small, with a book club looking at Agile project management via throughput accounting to Eli Goldratt’s classic The Goal. This led us to implement a programme of coaching, workshops and one-on-one sessions and we embedded specialists in various cross-functional teams. This in turn led to a shift in focus so that ideas and innovations for process change are coming from people on the ground rather than from Seth or myself. So, eighteen months into our three month experiment, we are just where we wanted to be, and it’s time for me to do something else.

In many ways it’s been a dream gig. I’ve seen too many change programmes fail due to management inertia or even active sabotage. DRW Trading is blessed with the kind of enlightened and open-minded management team that doesn’t exist in the real world. Working with the triumvirate of Seth as CIO, Derek Groothuis as Director of Software Engineering and Lyle Hayhurst as CTO has been a fun, enlightening, invigorating ride and I’ll be sad to leave them, as well as a lot of insanely talented folks. I expect I’ll still be visiting from time to time.

Open for business

I’m excited about the possibilities for Dan North & Associates. I will be retaining my independence and contrarian outlook. I promise not to run any course with the word “Certification” in it. My first public course is Faster Software Delivery at Foo Café at the end of August, and I’ll be announcing further courses and conference appearances as they shape up.

Drop me a line if you want to talk about working together. In the meantime I’ll be adding case studies and other reference material to the site over the coming weeks.

The article is now available on my site. Please let me know what you think about it in the comments below.

Is BDD the same as TDD? Yes. If you’re a programmer, and your entire team is programmers, and all your stakeholders are programmers and you have a single Subject Matter Expert embedded in the team. Which was true of the Chrysler C3 team and other early XP teams. (Take a look at the original C3 team: 10 world-class programmers and an SME.)

BDD is the same thing but for a broader audience: testers, analysts, project and program managers, multiple SMEs covering multiple interrelated domains. It’s interesting that the BDD == TDD crowd are entirely programmers and necessarily see the world through programmers’ eyes: Bob Martin, Ron Jeffries and others saying “BDD is just TDD” is true in the first approximation of “all delivery people are programmers.” As soon as that precondition fails, BDD becomes a different, bigger thing: it becomes the communication across all of those stakeholders to create a single coherent vision and deliver to that. This is the value of living documentation – and shows why you don’t need it until you do: if you’re in that small team of only-programmers you’ll be fine with only-TDD.

It can be argued that the team shouldn’t be anything other than programmers – in the polyglot roles of tester, analyst, etc. – in which case BDD and TDD again collapse into the same space. But in the kind of organisations and teams that gave rise to and shaped BDD we have to solve the more complex communication problems that are (by definition) beyond the scope of TDD.

I’ve been entirely embedded in small teams of programmers for the last couple of years which is why I’ve taken much more of a back seat in the BDD community. The likes of Liz Keogh, Gojko Adzic and Chris Matts have been where the action is with things like real options and specification-by-example, and folks like Aslak Hellesøy and Matt Wynne figuring out how to manage large numbers of long-lived tests at scale. Liz recently wrote about how shifting the language helps to clarify intent.

Kent Beck describes XP as trying to get business people and technical people on the same page, and I have the same goal for BDD. Don’t forget TDD is one small part of XP. BDD has grown to be broader than s/test/should/, because it’s trying to solve a broader problem.

JavaScript the Survivor

JavaScript has been around for a long time, and it has the scars and stories to prove it. As languages go it has more than its fair share of corner cases, incompatibility issues, frustrations and quirks. It’s an easy language to knock: it was hit with the same ugly stick as the other curly bracket languages, it’s more verbose than other dynamic languages like ruby or python, and if you can look at a piece of code and tell me what the value of the implicit this variable will be, well, you should be getting out more. (Ok, I’m being mean here. Nowadays I usually know what the value of this will be, but let’s just say it isn’t exactly obvious to a novice.)

JavaScript had a difficult childhood. It grew up in lawless neighbourhoods surrounded by gangs. It spent a lot of time listening to its parents fighting with one another about what they wanted it to be when it grew up. As any young language would, it tried hard to please its parents (and that barmy committee of uncles, and all the other random people trying to shape its future). As a result it suffers from what can only be described as behavioural quirks. Depending on which gang it’s hanging out with it will sometimes happily talk to you through its console.log, at other times refuse to say anything, and yet other times it will blow up in your face (but not tell you why).

It has collections, just like the other languages, but no sensible way of traversing them. Instead you are left with the delightful:

for (var key in map) {
  if (map.hasOwnProperty(key)) {
    var value = map[key];
    // right, now we can get some work done
  }
}

Now you see that var key at the top of the for loop? That’s not declaring a variable, oh no. It’s saying that somewhere else there’s a variable called key (right at the top of the nearest containing function, it turns out). Oh, and that key variable is visible all through that function, and any functions it contains, not just tucked away in the for loop. Brilliant!

So you see, it’s easy to knock dear old JavaScript. Google has even gone to the trouble of coming up with an entire new language for the browser to try to appeal to the Java- and C#-loving, class-conscious (ahem) programmer. But I think they are wrong to do that. All JavaScript needs is some love, and to be treated with a little respect. Sure every now and then it will crap on the bed, but it will also reward you with a wonderful development ecosystem if you make the effort to learn its little ways.

JavaScript grows up

During the browser gang wars, several things happened that started bringing peace and unity to browser-based development. JavaScript started to gain a reputation as someone who could smarten up your neighbourhood: it could provide all sorts of excitement and turn your boring old browser into a cool, asynchronous playground. However it still had its behavioural oddities, and the different gangs had virtually their own language for saying the same thing (as gangs do). Doing something in more than one neighbourhood often involved pretty much starting over.

However, a number of the other kids decided JavaScript wasn’t so damaged that it was beyond help. They realised the gangs were more similar than different and started studying their various quirks. Mediators began to emerge between JavaScript and the various gangs, so you could talk to a mediator and they would figure out what you meant. This was great, except it led to the forming of a new set of gangs. You could join the GWTs or the extjs, the prototypers or the YUIs. But once you’d chosen your path it proved difficult if not impossible to work with the other mediators. They all had a different idea about how to speak both with the gangs and with JavaScript itself. Some of them wanted to “protect” you from JavaScript’s perceived ugliness, and some even tried to pretend it was a whole other thing (say, class-based OO). Others were content just to talk to the gangs on your behalf and leave you to talk to JavaScript directly.

Then one day a new mediator called jQuery appeared on the scene. jQuery wasn’t like the others. Its designers figured that in the browser you mostly cared about exactly three things: handling events, manipulating the DOM, and talking back to your server, and that if they made those things really easy you could probably figure out the rest, including JavaScript itself. jQuery was game-changing in its elegance. (Don’t get me wrong, apparently inside the jQuery house is some pretty shocking JavaScript, but luckily no-one ever visits there.) It made use of familiar constructs, like using CSS selectors to identify elements in the DOM, just like you do. It allowed you to treat the DOM as the great big cuddly hierarchical data structure that it is, offering you the functional constructs of internal iterators, filters, chaining and so on. Suddenly the browser felt like it had been tamed.

In the meantime JavaScript had gone into therapy and emerged as ECMAScript, which meant that if you wanted JavaScript in your neighbourhood at least you had a good idea of how it wanted to behave, even if you weren’t going to let it. This would have been great except it was that barmy committee of uncles that were acting as the therapists, so things moved like treacle. (One of the uncles, Uncle Doug, has some interesting tales about just how slowly things can move). Meanwhile other events were unfolding.

The browser grows up

Google, the mighty advertising search company, had been slowly and successfully making inroads into browser-based apps with GMail, Google Docs, Maps and other cool technologies. They even have their own Marauders’ Map. Then a few years ago Google moved into the browser space. I’m not an industry pundit so I won’t try to predict what their business strategy was or is, suffice to say it looked like they saw the browser as the platform of the future, and decided they may as well be the dog as the tail.

They were already stretching the browser to the limits of credibility with GMail – a fully-fledged mail, calendar and contacts app in your browser. They had to invent technologies like Google Gears to provide client-side storage, and then persuade you to install them in your browser (which was a pretty easy sell since it gave you offline mail and document editing in your browser). So they created their own browser, called it Chrome, and then decided to do three very smart things in order to make a consistent, high-performance browsing platform as ubiquitous as they needed it.

Firstly they open sourced their browser so people could see what they were up to. This meant other open source efforts, in particular Firefox, could borrow from their technical innovation, which upped the bar for everyone. (Firefox wasn’t waiting around either. With each new release of the browser and its Gecko engine came improvements in rendering times and JavaScript processing speed, as well as better support for emerging standards in the various browser technologies.) Secondly they started to push very strongly for clearer standards across browsers, giving rise to the nebulous HTML 5. Certainly there are other major players involved in HTML 5, not least the venerable Yahoo where Uncle Doug lives, but Google’s involvement created a sense of urgency that was never there before.

Thirdly they realised the browser is more than just a place to render HTML and execute JavaScript. Google Chrome contains a full suite of development tools: a REPL to interact with the JavaScript in the current page, a DOM inspector to traverse the DOM and inspect CSS styles and how they got there (not the DOM that was loaded, but its current state after all your JavaScript manipulations), a network analyser to tell you which page elements are loading (and failing) and how long they took, basically everything the jobbing web developer needs to iterate quickly. Other browsers, notably Firefox, have these available as plugins like Firebug and Web Development tools, but Chrome gives you them out of the box.

Even Microsoft decided they needed to get on board with the new wave of HTML 5. First they started referring to the emerging HTML 5 standards as “tier 1 supported technologies” which was heresy to the Microsoft Old Guard. Then they quietly dropped Silverlight, which was their competing not-invented-here browser technology (and which of course only works on Windows). After about 100 years of IE6, they released IE7, IE8 and IE9 in rapid succession, each one faster and more standards compliant than its predecessor (although it’s fair to say that IE is still the red-headed stepchild of web conformance, but at least they’re playing the game).

Now HTML 5 is a huge slew of initiatives covering not only CSS, HTML and JavaScript standards but 2d and 3d graphics, full-duplex I/O with WebSockets (and half-duplex with EventSource), sounds, video… In fact if you stand back and squint you could be forgiven for mistaking the HTML 5 ecosystem for an entire operating system. Whose system language is JavaScript.

JavaScript on the server

The next piece of the JavaScript puzzle starts a couple of years ago with a young hacker named Ryan Dahl. He figured we were doing I/O all wrong on the server, and that in today’s multi-core, highly-concurrent world this was never going to scale.

Mostly we do something like this:

context = "today";      // a local variable
data = file.read();     // synchronously read data
process(data, context); // and process it

where the thread doing the read blocks until data has been read from the filesystem and loaded into a buffer. To give some idea of scale, memory is tens or hundreds of times slower than CPU cache (measured in orders of ηs), socket I/O is thousands of times slower than memory I/O (orders of μs), and file and network I/O is thousands of times slower than socket I/O (orders of ms). That means your thread could be doing literally millions of things instead of clogging up the place waiting for some data to come back.

An asynchronous version might look more like this:

context = "today";  // a local variable
file.read(          // ask for some data
  function(data) {           // callback is invoked some time later,
    process(data, context);  // still bound to context
  }
);

where the read function immediately returns and allows execution to continue, and at some arbitrary point in the future — when the data is available — that anonymous function is called with a reference to the context variable still available, even if it’s gone out of scope or changed its value since the call to file.read.

Now this is completely foreign to most server-side programmers. For a start the major languages of C#, Java and C++ are various kinds of useless at bindings and closures — and even Python and Ruby aren’t big on callbacks — so most server-side programmers aren’t used to thinking in these terms. Instead they’re quite happy to spin up a few more threads or fork and join the slow stuff. And this is just the first turtle: What if the process function itself is asynchronous and takes a callback?

However this model is bread and butter to your JavaScript programmer. The entire browser is predicated on a Single Event Loop: There is only one conch shell so if you (or more accurately your function) has it, you can be sure no-one else has. Your concurrent modification woes evaporate. Mutable state can be shared, because it will never be concurrently accessed, even for reading (which is why the whole DOM-in-the-browser thing works so well with event handlers). They are also aware that the price of this is to give back the conch as quickly as you can. Otherwise you bring the whole browser grinding to a halt, and no-one wants that.

So if you’re going to try to pull off this kind of evented I/O shenanigans on the server, well what better language to use than one where this is the core paradigm? Ryan looked at Google Chrome’s shiny new V8 JavaScript engine and asked: what would it take to get this running on a server, outside of the browser? To be useful it would need access to the filesystem, to sockets and the network, to DNS, to processes, … and that’s probably about it. So he set to work building a server-side environment for V8, and node.js was born.

Fast-forward a couple of years and node.js has grown into a credible server-side container. As well as the core container, the node.js ecosystem contains a saner-than-most package manager called npm, and libraries for everything from Sinatra-like web servers, database connectivity, handlers for protocols like IMAP, SMTP and LDAP. In fact I find most of my time is spent coding in the problem domain rather than trying to wire together all the surrounding infrastructure cruft. (Sometimes I find myself deep in a rat hole trying to figure out why a library isn’t working, but such is the way of open source. At least I can look at the source and add some console debugging.)

It uses native evented I/O on Linux and until recently would only run with an emulated I/O layer on Windows. Until Microsoft got involved. Yes, the mighty Microsoft is actively aiding and sponsoring development of tiny little node.js to try to produce equivalent performance on Windows using its (completely different) native evented I/O libraries.

The current stable branch of node.js is showing bonkers fast performance on both Linux and Windows, and only seems to be getting faster. (The fact that Google’s V8 team are big fans and are actively considering node.js on the V8 roadmap isn’t doing any harm either: they suddenly received a slew of bug reports and feature requests as the uptake of node.js pushed the V8 engine in new and unexpected ways.)

JavaScript everywhere

And it doesn’t stop there. JavaScript’s serialization form, JSON, is becoming ubiquitous as a lighter-weight alternative to XML for streaming structured data, and NoSQL databases like mongo are happily using JSON and JavaScript in the database as a query language. This means, for the first time, you can have the same JavaScript function in the browser, on the server and in the database. Just think for a moment how many times you’ve written the same input validation or data checking logic in three different technologies? I’ve not been using any database-side JavaScript in the work I’ve been doing, but it is an interesting proposition.

Tablets such as the sadly-but-hopefully-not-forever-demised HP TouchPad and the nascent Google ChromeOS tablets are using JavaScript (and node.js!) as a core technology. Instead of targeting a closed platform like iOS you can simply write a web app — possibly taking advantage of other tablet-specific APIs or event sources such as a GPS or accelerometer — and it will Just Work on your tablet device. Now that’s pretty sweet! Sadly the most open technology is available on the smallest proportion of devices, and vice versa, but you can still reach a lot of people just by writing a slick web application.

One interesting tangent is that a number of folks are considering JavaScript as a compilation target for other languages! (The implication being that JavaScript is so broken that you wouldn’t want to actually write it, but it’s ok as a kind of verbose assembler language for the browser.) The Clojure language has recently spawned ClojureScript, which is a compiler that emits JavaScript so you can run Clojure in your browser, and a new language called CoffeeScript has emerged, which is a sort of “JavaScript, the good parts with a Pythonesque syntax and a nod to Ruby.” It aligns very closely with JavaScript — you can see exactly how your CoffeeScript expressions turn into the equivalent JavaScript — but lets you get away with less syntax. I used CoffeeScript for a short while, and found that the best thing it gave me was an appreciation of how to stick to the good bits of JavaScript. I expect I’ll talk more about why I leapt into CoffeScript — and why I then decided to decaffeinate — in my next article.

And finally…

One thing I keep noticing is that by being this late to the JavaScript party, a lot of the lumpier language-level problems have been solved, and solved well. One example is a library called underscore.js that describes itself as “the tie to go along with jQuery’s tux.” It elegantly provides missing functional idioms like function chaining and the usual suspects of filter, map, reduce, zip and flatten, in a cross-browser and node-compatible way, as well as some useful methods on object hashes like keys, values, forEach and extend (to merge objects) etc. Similarly a library called async provides clean ways of chaining multiply-nested callbacks, managing multiple fork-joins, firing a function only after so many calls, etc.

As I said at the beginning of this article, I don’t remember feeling so empowered by a technology, in terms of being able to deliver full-stack applications, as I am by the combination of HTML 5 and server-side JavaScript. It’s an exciting time to be developing for the browser — and the web — and it’s never been easier.

This is something of a catch-up post, bringing you up-to-date with some of the things I”ve been up to and some of the topics I intend to be blogging and writing about.

Patterns of Effective Delivery

Probably the things I’ve been talking most about are Patterns of Effective Delivery, a topic I’ll be expanding on over the coming weeks and months, and how the principle of embracing uncertainty seems to underpin most of them. I’ve done a few talks this year about the patterns, culminating in a keynote at Øredev 2011 which was graphic recorded by the fantastic Nora and Heather at ImageThink:

Heather’s version

Nora’s version

Programming

One of the reasons I left consulting was that I realised I was moving further away from technical delivery and deeper into “pure” organisational change, and I really missed actual programming. I was still speaking at conferences and I noticed the stories and examples I was using were becoming less recent. Since joining DRW I’ve managed to strike a healthy balance between delivery and organisational change. Nowadays I get to code at work. With other people. It’s great! I’ve spent this year getting very excited about mixing up OO and functional programming, primarily with Python (I know, talk about late to the party!) and JavaScript (ditto).

JavaScript

It’s been an interesting mind shift going from a statically-typed, class-based, multi-threaded OO environment to a single event loop, dynamic, event-based language that treats functions as first class citizens alongside objects. I’m intending to write up some of the idioms and techniques I’ve been learning, discovering and struggling with, and why I think JavaScript is a wonderful and misunderstood technology. With libraries like jQuery providing DOM manipulations in the browser and underscore.js providing proper functional programming affordances on both client and server, JavaScript really is great fun to work with.

In particular I’ve been deploying server-side apps into node.js, the potentially game-changing server-side runtime based on Google Chrome’s V8 engine. Its inventor Ryan Dahl started with the premise that blocking I/O is wrong, on the basis that most I/O is usually thousands or millions of times slower than the other things your program could be doing. So by taking concerns like threading and I/O off your hands – in return for you writing callbacks or event handlers - the world becomes a happier place.

Python

I also intend to try sharing some of the joy I’ve found in programming in Python. Quite apart from the freedom of movement that thinking in functions can produce, I want to introduce you to list comprehensions, tuple-unpacking (a.k.a. destructuring), context managers, generators and coroutines – things that other, cooler languages are boasting about and that Python has quietly been getting on with for years.

I’ve started describing Python as runnable pseudocode: it’s become the language I carry around in my back pocket when I want to introduce non-programmers to the power of automation (displacing Ruby, which had been living there for the best part of a decade). It has the ability to quell fear and induce smiles in a way I’ve rarely seen in other languages. (I vaguely recall feeling the same about the magic it variable in HyperTalk many years ago.)

Organisational Change

My current gig involves working with all the non-programmer IT folks in a medium-sized trading firm, introducing ideas from Lean Operations and Theory of Constraints. We’re doing things like moving the people to the work, identifying and mapping value streams, measuring throughput and effectiveness rather than effort and activity – ironically all the things I was trying to introduce to my clients when I was a consultant!

So far we’ve had some encouraging successes. We’ve taken something like two thirds off the time it used to take to purchase and commission a new server, simply by focusing on reducing the >90% waiting time rather than trying to optimise the <10% of activity that was already pretty slick! (In my experience those numbers are fairly typical: when you first apply value stream mapping to a traditional process you often find well over 90% f the cycle time is lost in hand-offs and waiting.) The manager who owns the server commissioning process managed to chop one third off the time just by making two simple changes to the process. Talk about quick wins!

But the best part of this gig is that everyone is open to trying these ideas. Not just the people on the ground or their line managers, but right up to the CIO and CTO, people are not only receptive but are actively exploring ways to enable their organisation and challenge existing thinking. Most of my time and energy in Big Enterprise consulting involved trying to influence senior managers to think differently and embrace some of these ideas. It was exhausting. These days I find I can just get on with it: When I push on a closed door it usually flies open at the slightest touch. I’m still adjusting to that after nearly two years!

One of my plans is to write up some of my experiences introducing organisational change in this environment. I remember reading Esther Derby and Johanna Rothman’s excellent Behind Closed Doors, a management book that uses an experienced manager as a vehicle to illustrate how good leadership and team management can work. Many management books spend a lot of time advising on how to tolerate, subvert or otherwise handle toxic organisations. Esther and Johanna’s approach on the other hand was to invent a great manager in a grown-up organisation and use him to describe the art of the possible. I’d like to try something similar with organisational change: I haven’t seen much literature about how to operate when everyone wants you to succeed. I promise you, it’s a very different experience!

I’d like to thank…

This has been an interesting year for me in terms of meeting or reconnecting with people who have been influencing my thinking. I was lucky enough to encounter Benjamin Mitchell at QCon London earlier this year. He’s an irritatingly thoughtful and insightful guy who doesn’t let me get away with sloppy thinking. We meet for all-too-infrequent coffees and argue. He’s probably unaware of this but he’s been one of my biggest influences over the last year, and I’m especially grateful for him introducing me to the work of Chris Argyris and his Ladder of Inference.

A number of my work colleaguse at DRW have been a huge influence on my thinking over the last couple of years, not only on a technical level but also organisationally and personally. I’m not going to single them out because they’ll just point at me and laugh, but they know who they are.

Liz Keogh and Chris Matts continue to amaze me with where they’ve taken BDD, Gojko Adzic published Specification by Example, which gets me nicely off the hook for writing a BDD book, and I’ve been delighted to see Aslak Hellesøy and Matt Wynne’s Cucumber book make its way into print. It’s a marvellous read and you should go and buy it. Speaking of BDD, I owe a big thank you to Wendy Devolder and the other fine folks at SkillsMatter for hosting various in-the-brain sessions and organising the BDD Exchange (including getting all the videos online seemingly within minutes of the talks finishing). There are many others whom I’ve been learning from and exchanging ideas with during the year, so this is by no means an exhaustive list.

Others have been busy too. Arialdo Martini has translated my Introducing BDD article into Italian, and is currently working away translating some of my other articles. I’ll keep you posted on his progress. There is also a new Bulgarian translation of What’s in a Story? that I want to share with you, but I’m still trying to track down a reliable URL (the one I have keeps falling off the Internets).

Well I think that brings us pretty much up-to-date. If this is still the latest post on my blog in three month’s time, you’ll know I’ve failed to start writing regularly. Hopefully it won’t be.

The aspect I want to focus on in this article is domain language, because there’s a failure mode I encounter surprisingly often, which seems to have a common root cause.

Here’s an example:

Story: User logs in

Scenario: User with valid credentials

It seems innocuous enough. It’s a scenario about logging in, a common feature on a web app, but let’s take a look at the vocabulary in use here. We can see language from a number of different domains:

• Words like unauthenticated, valid, credentials are from the domain of security, more specifically user authentication.
• Words like name and password are from the subset of that domain concerning password-based authentication.
• Then we have words like enters, field and button. These are from the domain of UI widgets.
• Finally we have login page and welcome page. These are from the domain of web assets.

Imagine we’ve delivered this feature and the scenario works as described. Now what happens if we go from password-based authentication to OpenID, or a centralized authentication system (CAS) model? This would obviously break the scenario. The requirement for a password becomes redundant so we would naturally expect to have to rethink the scenario. Alternatively what if we change the UI so the name is now selected via a drop-down list or a radio button? The scenario breaks again, because there is no longer a Name field to enter a value. And again, what if we change the site navigation strategy and decide that a successful login should take you to the Dashboard page, bypassing the Welcome page? Once again we would need to reword the scenario.

If you are thinking this is starting to look like a bit brittle, then you are right. We’ve made the mistake of combining too many domains into a single scenario – an indicator that we are overspecifying something – which makes the scenario vulnerable to change along a number of axes. We’ve created accidental complexity because we lost sight of the intended audience of the scenario.

Unpacking the domain

Let’s think about this for a moment. The intent of the scenario is that a user with valid credentials can gain access. That entire sentence lies in the domain of user authentication, so we could ask someone from that domain whether the premise of the scenario is correct. (Are valid credentials enough? Should we also require a physical token? Can credentials expire?) In general, any domain will map to a single subject matter expert, who we can call the stakeholder or representitive for that domain. In fact it’s a circular definition: a domain is simply the “subject” in the phrase “subject matter expert,” so for any given domain we should be able to identify a single stakeholder who represents that domain.

As an aside, this raises some interesting questions. What if you are writing scenarios in a domain that no-one seems to care about? (You can tell by watching their eyes glaze when you talk about it.) A lot of what we traditionally call non-functional requirements can fall into this category. For instance, most non-technical people aren’t interested in networking terms like latency, throughput or packet loss, but they might perk up when you start talk about sluggish response times or requests going missing. You can use the glaze test as a heuristic to know if you are talking to the wrong person – or using the wrong language. Similarly, if you can’t the answer the question “who cares about this requirement?” with an actual name, you either have a redundant requirement or a missing stakeholder.

But back to our login story. Every domain we include in a scenario potentially pulls in an additional stakeholder, whose requirements or priorities might change at any time. So to avoid a scenario becoming brittle we want to involve the minimum number of stakeholders we can get away with. The simplest useful scenario should involve exactly two domains – and therefore only two stakeholders – namely the domain of the scenario title (the problem domain, or the what) and the domain of the steps (the solution domain, or the how). We can’t have fewer than two domains. If we exclude the problem domain we aren’t explaining the value, or the intent, of the scenario. If we exclude the solution domain the scenario can’t describe the behaviour we want from the application. But any additional domains are likely to provide unnecessary constraints or noise, and make the test brittle. (When I talk about the solution domain, I mean the “outermost” solution domain – the solution’s interface if you like – as opposed to the domains of any of its implementation details.)

The what of our example scenario is logging in with valid credentials. The how is using a name/password pair to validate the user. The details of the how with its button clicks and whatnot aren’t adding any value to describing the capability we want. So maybe we could convey the same intent by rewording the scenario like this:

Scenario: User with valid credentials

Now we have exactly two domains, namely the “what” of user authentication (words like unauthenticated, user, credentials) and the “how” of web-based security (words like restricted asset, submits, redirected, content). If either of these domains changes then we would expect the scenario to change too. If we decide valid credentials are no longer enough we’ll probably need to add some more steps. Or if we decide to turn the app into a thick client then there will no longer be a login “page” so we might need to change that step to be a “modal dialog” or “screen” or something else thick-clienty.

Notice how we don’t explicitly say what it means to “submit valid credentials.” We’ve pushed this down into the implementation of the step. If we were to change the authentication model, say from name/password to OpenID, then the scenario would break, which we want (the implementation of the how has changed) so we would change the implementation of the step “the user submits valid credentials” to provide an OpenID URL rather than name/password pair, say.

But the wording and sequence of the scenario itself wouldn’t change because the intent is still the same and the behaviour is still the same – it’s the implementation of the behaviour – corresponding to the step’s implementation – that has changed, and that shouldn’t affect the meaning of the scenario.

Chunking – or the myth of “declarative”

Neuro-linguistic programming (NLP) describes a technique called chunking, that’s useful for either solving problems or creating options. For any statement, you can chunk up by asking “Why..?” or “What for..?” questions and chunk down by asking “How..?” questions. The further you chunk up, the broader your perspective becomes, and the further you chunk down, the more detailed. The power of chunking comes when you start to chunk sideways, by asking “How else..?” questions.

For instance with our login scenario, we could chunk up with: “What are you logging in for?” Perhaps it’s to restrict access to premium content, or for regulatory reasons. If it’s the former we could chunk sideways with: “How else could you restrict access?” Perhaps we could filter on IP or MAC address, or use cookies, or single sign-on. Then we can chunk down again to consider solutions: “How could we filter on MAC addresses? What new security issues could this raise?”

Once you realise you can ask “What for?” or “How?” at any layer of abstraction, concepts like “declarative” or “imperative” suddenly become relative. Any layer is the what of the layer below, and the how of the layer above. SQL is often described as a declarative language: you describe what you want to select but you don’t tell the database how to find it. However, the statement select employee_id, salary from employees where salary > 100000 could equally be considered imperative, a command, if the “declared” requirement is “find all employees paying higher rate tax” (tacit knowledge: higher rate tax kicks in at 100k), which itself is an imperative implementation detail if the original request is “establish our income tax liabilities by rate type.”

Be deliberate in your use of domain language

So tying this all up, when writing your scenarios keep in mind that you are writing them for two audiences: the person the feature is for and the person implementing it. Check the wording to see if you can spot anything from neither the problem nor the solution domains. If you find you are using language from outside those domains, you might be over-specifying the implementation or specifying unnecessarily broad requirements that mix concerns.

If you really care about how the behaviour is implemented, you should probably be specifying that elsewhere in a more fine-grained story – in other words chunking down to provide more detail – that won’t be interesting to the audience of this one. If not, you might want to push the detail down into the implementation of the steps.

The overriding message I got was that without providing evidence and concrete examples one way or the other, it was all a lot of noise about terminology. One of my intentions this year is to expand on some of the implications of Deliberate Discovery, which fit the bill nicely in illustrating programming-as-a-trade, so my next article is the start of a series of posts along that vein. I’ll let the evidence speak for itself and you can draw your own conclusions.

I’ve been very grateful for the feedback and comments, and the dialogue on Twitter. I didn’t spend as long as I might have editing and cleaning up the article, largely because I’d been tinkering with it since before Christmas, rewritten it twice, and decided it was better to publish early and respond iteratively to feedback. Typical – go out without your best underwear and you get run over by a bus.

The critical feedback generally fell into the following categories:

• How dare you! You obviously know nothing of which you speak. (Well, you decide)
• You obviously don’t / have never / no longer code / care about programming. (Not true)
• You’re using a pretty crappy definition of “craft” to put forward a bogus argument. Here’s mine / Wikipedia’s / my dad’s / a selective entry from a dictionary’s. (That’s not the point)
• You shouldn’t/can’t compare programming to building / plumbing / fine art / martial arts. (See “On metaphors” below)
• I’m angry and important so I’m just going to poke you with a stick rather than do any actual thinking. (Meh)
• Where’s your evidence? I say “craft,” you say “trade,” show me evidence it’s a trade. (My next few posts)

It seems this is a sensitive subject area so I’ll be editing more carefully from now on to try to pre-empt some of the more obvious critiques I left myself open to in the previous post.

On value

If I gave the impression that I wanted the cheapest or quickest cobbled-together solution then I certainly didn’t mean to. I want the best value software, something that’s fit for purpose in this specific context, to solve this specific need.

I’m not going to mandate test-driving anything (which is a huge about-face from what I was saying a year ago), unless it will help. Copy-and-paste is fine too. (Before you go all shouty at me again, hold off until I blog about the benefits of copy-and-paste, as it appears in a couple of patterns I’m calling Spike and Stabilize and Ginger Cake. You might be surprised.)

I expect a programmer to apply the appropriate amount of rigour, discipline and excellence to any situation. They should be as comfortable hacking together a disposable script as test-driving a complex business calculation or mathematical algorithm, and they should have a good idea – based on experience – of when to adopt each of these attitudes. This to me is simply good, professional workmanship. (More about this in my next article.)

On metaphors

I shouldn’t have to point this out, but based on some of the feedback it seems I need to. Metaphor or analogy highlights a particular aspect of a situation. If I say something is as calm as a millpond, I don’t also mean to imply that it is wet and has lily leaves floating on it. (Yes I’m aware this is a simile rather than a metaphor.)

I used the example of jiu jitsu because I studied it long enough to qualify as an instructor so I know a bit about it. I remember the awe I felt when I witnessed the calm, grace, power and control of two black belts performing their nage-ne-kata – the same thing that I stumbled clumsily through from time to time – at a national competition one year. They got a standing ovation. It was beautiful. Programming is not a martial art.

I used the metaphor of negative space to illustrate that the point of programming isn’t the software – it’s the information flow or utility that the software enables. Programming is not fine art.

On certification

I don’t agree with, support or condone in any way classroom- or course-based certification. It’s a ridiculous money-making charade. (Yes, Scrum Master certification, I’m looking at you.) An inexperienced bully of a manager with a certificate is still an inexperienced bully of a manager.

I was attempting to open up a debate on how we might possibly recognise another’s skill and experience if we don’t a priori know them. This isn’t just a theoretical debate, it’s central to good hiring – or engaging outside help.

I have a lot of regard for experience-based and evidence-based qualifications. The way Dr. Patricia Benner applied the Dreyfus model to nursing in the US, for instance, completely reinvigorated the service. The sheer amount of hard graft and knowledge that most MBA-holders I know put into their studies is testament to how hard they worked to get one (and that they didn’t even bother trying until they had enough experience to attempt it).

I guess I’m saying any qualification worth its salt should enable me to see a curve of, or at least differentiate between, people-who-can and people-who-can’t (rather than just showing people-who-paid). A common refrain was that our industry and technologies change so rapidly that there wasn’t a syllabus or set of course content that could be sensibly assessed. I think I agree with this.

Some of the suggestions that came back were:

• Look them up online! If they’re any good you should find evidence of that in terms of their own online output or other people’s testimonials. I really like this, especially nowadays, but the counter doesn’t hold: I might overlook some great-but-quiet individual if this becomes my core criteria.
• Have certification. See above
• Become (literally) a journeyman, where you rove from company to company, working for a roof over your head, building your reputation as you go. I love this idea but it takes some logistics.

Basically this debate is going to run and run. And I’m glad about that. It’s a really hard problem.

On open source

I consider contributing to open source projects an admirable use of time and a good way to practise your programming. However the average quality of most OSS I’ve seen over the years is bloody shocking. It’s utility, however, is undeniable.

One of the most widely used pieces of OSS is the venerable JUnit. Ironically this was thrown together by two guys on a plane on their way to a conference, who had some time to kill on a long flight. Partly this was so one could learn Java and the other TDD. The fact that the two guys were Kent Beck and Erich Gamma demonstrates that a) two smart and pragmatic enough programmers can put together pretty much anything under any conditions in a few hours, and b) you can get enormous utility without having to rigorously test-drive everything. (Although in this case I understand they did use TDD - Thanks Martin Fowler for the corrections.)

As another example, Joe Walnes, my examplar of the incredibly-talented-and-mildly-amused kind of software craftsman (see On craftsmanship, below), had one of his many open source projects rejected by a code repository’s review panel because it was “too simple – there was no point making it into a library.” Turns out it proved quite popular.

Open source software is as variable as any other software. The fact that it’s open source doesn’t say anything about its quality. It doesn’t mean it is not written or hosted by muppets (or genuinely talented people, for that matter – its open source-ness is simply an orthogonal concern). It’s most certainly not an “apprenticeship model,” but it’s a great way for people to read other people’s code (something we are really poor at as an industry), share knowledge and share the wealth of the utility of software.

On productivity

I don’t have any scientifically-valid numbers for software productivity. I have a ton of informal, empirical evidence that a small group of talented individuals can be insanely productive, even compared with other teams of talented individuals. I’ve been fortunate enough to spend the last year living it (and it’s nearly killed me trying to keep up!).

The people I had in mind when I mentioned hundreds-of-times productivity have done things like standing up an entire soup-to-nuts electronic trading stack, from user-facing visualisations of real-time trading data through to fully-automated trading applications – in a few weeks. These are not standalone applications. They talk to (really complicated) financial exchanges, internal back office settlement and reporting systems, and all the usual corporate messaging infrastructure you would expect to see in a medium-sized firm. To use the words of Lewis Carroll, this is, of course, impossible. I’ll see your factor of 10 and raise you 100. It’s not common. In fact it is extremely unusual. But it does exist.

On programming as a trade

Maybe I should have said “programming has far more in common with a trade than it does with a craft.” As with any analogy, of course there are aspects of craft in programming. There are also aspects of craft in plastering or electrical wiring. I can spot the effects of well-executed, skilful workmanship even if I’m not qualified to rate the work itself. There is just much more of an affinity with trade-ness. In the first instance software should be practical and fit for purpose. I want effective, I don’t want fancy.

The guy who fitted my satellite dish ran the cable incredibly carefully around door frames, through walls, hugging the tops of skirting boards – it was virtually invisible – because “I’ve got a wife too, mate.” My house is old. Many attempted refactorings haven’t worked out and have ended in at least partial rewrites. (Uh oh, a metaphor again. I know programming isn’t DIY.)

People want software because they want to solve a problem, not because they like the notion of software. Some software practitioners risk losing sight of that. That’s all.

On craftsmanship

Some people really care about understanding “the Thing” that we do when we try to get computers to help us do stuff. They think about it, write about it and exhibit it in their work and their behaviour. (Thanks, Michael Feathers, for a thoughtful and insightful post.) These people are sometimes called craftsmen. I admire these people.

Some people self-identify as “rock stars” or “software craftsmen” in an unfortunate attempt to differentiate themselves from regular jobbing professionals, whom they regard as less talented or valuable than themselves. I don’t have much time for these people.

Some people are just really bloody good at writing software. They rarely blog, they don’t talk or write about software. They just love coding, solve difficult technical or domain problems in innovative and highly skilful ways in a fraction of the time of us regular mortals, to deliver real benefits. They learn a new tool or technology by using it in real work situations and figuring it out as they go. They think all this navel-gazing, posturing and chatter is a bit silly. I admire these people enormously! Some of them like to tease me. I’m ok with that.

Some people care so much about their profession (I’m using the term in a broad sense here to mean what-they-do-for-a-living) that they choose to help other less experienced practitioners in their field grow and develop their skills. They use their own time to write and freely publish material, or provide workshops or classes, out of a desire to raise the bar across the industry. They might go travelling, not only to develop their own skills but to share their skills and knowledge as they go. (And it makes a great odyssey story!) Sometimes they do this enough that people recognise it and call them out as software craftsmen. I think I admire these people most of all, because they are inspiring the next generation of programmers to think like that too.

Software Craftsmanship risks putting the software at the centre rather than the benefit the software is supposed to deliver, mostly because we are romantics with big egos. Programming is about automating work like crunching data, processing and presenting information, or controlling and automating machines.

Non-programmers don’t care about the aesthetics of software in the same way non-plumbers don’t care about the aesthetics of plumbing – they just want their information in the right place or their hot water to work. (Although it’s fair to say they appreciate decent boiler controls.)

Motivation for software craftsmanship

It would be great if programming were a proper profession, but it isn’t. A profession has a structured model for advancing through levels of skill and ability, be it studying for a law degree and articles (working for a legal practise) or the years of undergraduate and medical training a doctor undertakes before specialising. The latter has clearly-delineated ranks, from junior doctor, via a brutal regime of 80-hour weeks, to consutant.

Conversely there is no minimum entry requirement for programming. Some people naturally have a flair for it (two of the best programmers I know never went to college), some teach themselves out of books, others just tinker until they get something working. A programmer’s skill and ability is only as good as their personal reputation: there isn’t an accepted, transferable ranking like there is in a “proper” profession.

To illustrate how low the bar can be, the mission statement of Visual Basic (whether explicit or not) was to democratize programming. Anyone who could drag and drop controls around and understand a modicum of technical stuff could throw together a solution to a problem with relatively little effort. Wizards and other utilities enable power users to create sophisticated spreadsheets in Excel with no practical programming knowledge. Language guru Erik Meijer has spent the last few years talking about democratizing the cloud, i.e. making it as easy to string together online services and utility computing as VB made it to string together other applications. In other words, there are people actively working to lower the already low barrier to entry for programmers. (These are Microsoft-specific examples, but then Microsoft has done more than most to dumb down programming, or make it accessible to the masses, depending on your point of view.)

The IT industry is relatively young – only a couple of generations old in fact. (As an experiment, go and find out how many of your coworkers have a parent who worked in IT. See?) It is also something of a gold mine. Compared to a lot of industries it is relatively well paid indoor work with no heavy lifting, and remember the thing about no minimum entry requirement? This means entire economies have grown up looking at IT as a numbers game: if you throw enough bodies at a problem you should be able to make it go away, and if you can get bodies for cheap enough (yet still relatively well paid compared to the alternatives) then you can throw a lot of bodies at a problem.

I spoke to someone once who told me he was currently engaged in a public sector project to provide online tax credits for builders. As a builder you would have an account online where you would log in and buy tax credits that were offset against your end-of-year tax bill. In other words, it was a site that had user accounts and a (third party) card payment facility, and not much else. This project had 400 programmers working for three years. That’s more than a person-millennium of effort. For a web app! That I’m paying some consortium for out of my tax money. But that’s a rant for another day.

So from a purely demographics perspective we can see that the vast majority of people in the IT industry are there because a) it’s a well paid alternative to other white collar office work or even manual labour, and b) there is no incentive to make it anything other than a commodity numbers game.

Then there are the others. The minority of people who genuinely love programming and choose to excel at it. They understand software development is a skill, in fact a whole portfolio of skills: understanding and modelling a problem domain, understanding programming languages, libraries, paradigms and idioms, choosing which to apply in a given situation, learning and understanding algorithms, mastering the “path to production” (build, deployment, release), monitoring and availability, process automation, Lean theories of supply, production and product development, utility and cloud computing, concurrency and parallelism, I could go on. Those guys want a way to differentiate themselves. The oft-quoted figures of tenfold increase in productivity of expert versus novice programmers are wrong by orders of magnitude in my experience. A really great programmer (and I’ve been lucky enough to work with a handful over the years) can out-perform a doing-it-for-the-money programmer by orders of literally hundreds, delivering in hours or days what would take an average developer weeks or months.

How are those guys supposed to differentiate themselves? And how can they help others in the industry who have a real love and appreciation for the software they write? We need some sort of apprenticeship model, and a way to identify masters, both to apprentices and other masters. That sounds like the sort of model that craftsmen use. And it also appeals to the average alpha geek’s romantic streak, possibly incorporating a system of secret signs and handshakes.

The thing is, at one level software can be described by the utility it provides. It doesn’t matter how ugly it is under the hood as long as it delivers the goods. A programmer can show beautiful software to another programmer, but that’s where the appreciation stops for software per se.

Seeing the negative space

My wife is an artist, and one of the things she studies is negative space. This is the space between the obvious shapes or objects in a picture, as opposed to the objects themselves. An understanding of negative space is critical to being able to faithfully render a composition. As the model sits sideways on, notice the curved triangle bounded by the upper and lower arm and the side of the body. See how the space between the front of the face and the window frame in the background is nearly rectangular and quite narrow, and see how the outline of the nose creates a dent in the left-hand side of the rectangle. Look at the chin, actually the space below the chin, to observe the angle between the bottom of the jawline and the front of the neck. And through the window, notice the colour of the sky in the space between the winter tree branches. It’s different from the grey-white cloud above the trees. All of these shapes and colours – the space between the obvious stuff, the negative space – these are the shapes that determine how faithfully you represent the image on a canvas. But mostly you don’t notice them – especially in a good painting.

So what does this have to do with software? Well it seems to me the most succesful programmers I’ve encountered don’t craft software; they write software in order to move information around, in order to get something done. Information is the real deal – the software just defines the space that it moves around in. For those programmers, success is about getting information from point A where it’s currently languishing to point B where it’s going to actually be useful, as quickly and effectively as they can. Success in a UI is about rendering or capturing exactly the information that will be useful – no less and certainly no more – in a succinct, obvious way. The software is incidental, a detail, hidden away in the wings, and it is ultimately entirely disposable.

Why programming is not a craft

With a craft, the product has intrinsic beauty in its own right. A cathedral is really a big hut for people to meet in and worship. Make it from stone by all means, so it lasts longer than a wooden hut, but why all the fancy decorative stuff? Of course it is there to engender a sense of majesty and wonder, and cause us to engage the part of ourselves that appreciates beauty and magnificence, so we enter the cathedral reverently and humbled, ready to worship. What makes it a craft is the work above and beyond its basic utility to give it intrinsic, aesthetic beauty.

If I use the same stone to build a traffic bridge over a railway line, I care about its utility and structural effectiveness. If it’s a good, simple bridge I won’t even notice it’s there! We don’t notice most of the engineering that goes into the roads we travel on or the railways we use until something goes wrong! (There’s a really cool word for this phenomenon – when you only become fully aware of something when it fails – but I can’t recall it right now.)

There is a difference between the mindset of a master stonemason sculpting the expression on the face of a gargoyle and someone using the commodity blocks that make up a multi-storey car park. In the latter case the last thing I want is someone’s “personality” causing some of the blocks to be different sizes and no longer interchangeable, never mind the added expense of having someone manually hew the stone rather than using machine tools. In the former the stonemason’s attitude is indulgent. He is putting his signature (and his ego, and his reputation) into this magnificent representation of Hell’s best. If you just wanted an oil-pouring spout you could get one from a DIY store.

Software practitioners – especially, ironically, the good ones – often lose sight of this. They fall in love with the software itself and start thinking of themselves as craftsmen of software.

The easiest qualification in the world

So here’s my concern with the idea of Software Craftsmanship. It’s at risk of letting programmers’ egos run riot. And when that happens… well, the last time they went really nuts we got Web Services, before that J2EE. They convinced the British government that they wanted an uber-database to store Everything Ever About Everyone. You see where I’m going?

When I looked at the Software Craftsmanship Manifesto, something inside me died. (Something else inside me burst out laughing and told the other thing to stop taking itself so seriously, so we’re all better now.) Apart from being, well, bland, it is just so self-evidently an oxymoron. If its message is that we need to think of ourselves as software craftsmen and that this is somehow special, then what a spectacularly easy bandwagon it is to jump on.

Do I need to demonstrate any kind of skill? No. Any specific credentials? No. Any kind of experience working in the field? Nope (and as the Pragmatic Programmers are happy to remind you, ten years experience is very different from one year repeated ten times). In fact, all I have to do to associate myself with Software Craftsmanship movement is to fill in my name on the website. Woohoo! I’m now associated with the Manifesto of Software Craftsmanship! Is no-one else seeing the irony here?

The (original) travelling salesmen problem

Back in the day – while the West was busy being Medieval – Japanese warlords used to hire bands of mercenaries called Samurai to, well basically beat up other bands of Samurai. Your life expectancy as a Samurai warrior was usually pretty low, and dropped dramatically if you came off your horse. In such times the martial art of jiu jitsu was born. The premise with jiu jitsu is that you have come off your horse in battle: you are unarmed, outnumbered and your opponents have both a height and weapons advantage over you. You’re pretty much stuffed unless you can pull some cool moves out of the bag.

As Samurai moved around, hiring themselves out to different warlords, they would want to join the local martial arts school to continue their training. Now you can imagine, if you learn jiu jitsu in one dojo up to a certain standard (we’ll call it a black belt) and then turn up at another school, it’s pretty tricky to get any kind of calibration without involving some life-threatening fighting with the new kid. And without calibrating, you can’t know whether someone is any good.

Ever resourceful, the jiu jitsu schools developed a sort of dance, called the nage-ne-kata, comprising five groups of three techniques. For each technique you both execute the technique and have it done to you, to demonstrate not only your fighting skills but your falling and rolling abilities. All senior students would learn the nage-ne-kata, and if they went to a new school, the sensei would ask them to demonstrate it for them. It is still taught today in some ju jitsu schools.

You can tell a lot about someone’s jiu jitsu from their nage-ne-kata: how they stand, how they hold their stance, how they move, how graceful they are, how well they understand balance, how well they respect their uki (the partner), how well they roll and fall (and how quickly they get up!), whether they favour one side over the other, their spatial awareness. Just from one little dance.

Let’s be craftsmen then

So, back to the plot, I’m reassured by the kinds of people I see involved in the Sofware Craftsmanship movement, the likes of Kevlin Henney, Bob Martin, Corey Haines, Glenn Vanderburg. These are pragmatic, sensible, down-to-earth and above all genuinely humble people. My concern is that the glamour is overtaking the intent, as evidenced by the sheer number of people wishing to align themselves with the Software Craftsmanship movement.

I would love to see someone rewrite the Software Craftsmanship Manifesto in terms of getting results and delighting customers. I don’t want “steadily adding value,” I want “amazing their customers every day!” Software craftsmen should be egoless, humble, with a focus on the outcome rather than the code or the process. I’d like a call to arms to stop navel-gazing and treat programming as the skilled trade that it is.

No-one wants your steenking software – they want the capabilities it gives them, and they want those yesterday. (I’ll specifically exclude the User Experience folks. Theirs is genuinely a world of aesthetics and human understanding. They, to use a bricklaying term, do the face work – the fancy stuff people actually see.)

Maybe there should be a Software Craftsmanship Council, that confers membership in a collegiate yet transparent way, in just the kind of elitist model that desperately upsets the folks who have systematically dumbed down the education system in the UK (but that’s another rant for another day). You only get to join if those already there believe you will be a credit to the notion of Software Craftsmanship, just like Ph.D. viva boards do today.

A truly skilled programming team can deliver amazing business results in insanely short amounts of time. Let’s go after some of that! I want your experience. I want your knowledge. I want you to show me “the simplicity the other side of complexity,” to quote Oliver Wendell Holmes. It takes a real expert – a real craftsman – to see the elegant simplicity buried away inside the mess we call enterprise software, for instance, and tease it out.

Calling programming a trade takes nothing away from the desire for professionalism, experience and expertise. In the same way I want an expert electrician wiring up my house rather than a cowboy, I want an expert programmer enabling my business. What I don’t want, however, is a prima donna plumber who insists on talking about the elegance, beauty or art of plumbing, or who insists that I appreciate the aesthetic beauty of his joinery, or will “only work with other rock star plumbers, who only practise copper-driven plumbing.” The best software should be understated and unobtrusive (as, maybe, should be the best programmers). I don’t want to hear the clanking of information as it rattles from one poorly-implemented system to another, through ill-conceived interfaces.

So what now?

While I was formulating this article I made a few comments on Twitter about it and I received some interesting responses. One person suggested a programmer is a craftsman not unlike a silversmith. I would argue that the value of silver or gold jewellery is intrinsic to the jewellery. I don’t use the item for anything other than for others to admire its beauty (and by association my significant-yet-understated wealth and taste).

If we’re going to use gold or silverwork as an analogy it would be in something like a speaker cable, where the properties of gold contacts improve the quality of the sound. In this case, however, I want the gold connector to be identical to a cheaper one. If the cable manufacturer wants to demonstrate their aesthetic abilities, I’ll be a lot happier if they do it by making the cable look fancy than by messing around with the shape of the connector!

If you’ve read this far, then thank you. Now here’s what I want you to do. I do think there should be a Software Craftsmanship Manifesto, but not the thing that’s currently out there. I think it should be a call-to-arms, feisty, opinionated, brash and everything that a good manifesto should be (I’m channelling Kevlin Henney here). I also think there should be a way for passionate, skilled programmers to differentiate themselves from the mainstream commodity bodies, and also to recognise one another, and demonstrate their value to potential employers. What could that be, and how could we make it work?

As a buyer of software solutions, wouldn’t you want to know your systems were being built by master craftsmen rather than day jobbers? You’re paying for this and you deserve some kind of reassurance. Let’s figure out how to provide it.

Our street lamp is the Planning Game, which involves writing Stories and Estimating, using Planning Poker or other Estimation Techniques (everything in caps appears in the Agile Literature, and so has been deemed Official).

I suggested we are failing to use the planning time effectively, and that we should be devoting the time to finding out as much useful stuff as we can while everyone was in the same room, and I called this Deliberate Discovery. Marc McNeill commented: “Deliberate discovery. As opposed to accidental discovery? Or any other sort of discovery? Why add the extra word ‘deliberate’?”

Accidental discovery

If you do something a bunch of times, you will learn more and more about it. Probably. Potentially. Maybe. Ok, if you do something a bunch of times and unexpected things happen you will learn. If you just do the same thing over and over again, you will probably get better at performing that sequence, but you won’t learn anything new. The pragmatic programmers describe this as the difference between ten years’ experience vs. one year’s experience ten times.

Learning comes from experiencing the unexpected. As the saying goes: “Good judgement comes from experience. Experience comes from bad judgement.” When you receive an unfamiliar outcome you have to alter your model of the world to accommodate it (or dismiss the encounter as a fluke and only seek out reinforcing data – this is known as confirmation bias and is a great way to not learn). In Zen teaching this moment of enlightenment – of evolving your model of the world – is known as satori, and Zen students strive to induce these moments of satori by the use of koans. Similarly, the Dreyfus model of skills acquisition describes an Advanced Beginner as someone who is starting to put rote-learned rules into context – understanding where they do and don’t apply. This again can only happen when the learner experiences situations outside what they already know (which is why artificially constraining Best Practices can stifle learning).

This implies the student can accelerate their learning by actively seeking out encounters where they are likely to learn. This is the difference between accidental and deliberate discovery.

“Learning is the constraint”

Liz Keogh told me about a thought experiment she came across recently. Think of a recent significant project or piece of work your team completed (ideally over a period of months). How long did it take, end to end, inception to delivery? Now imagine you were to do the same project over again, with the same team, the same organisational constraints, the same everything, except your team would already know everything they learned during the project. How long would it take you the second time, all the way through? Stop now and try it.

It turned out answers in the order of 1/2 to 1/4 the time to repeat the project were not uncommon. This led to the conclusion that “Learning is the constraint”.

Edit: Liz tells me the thought experiment and quote originated with Ashley Johnson of Gemba Systems, and she heard about it via César Idrovo.

If we assume the only difference is that the second time round you have learned about the problem, this would suggest that the biggest impediment to your throughput was what you didn’t know. That’s not to say you spent all the extra time busily learning stuff. Heck that’s fun and what’s more it’s even useful! More typically you probably spent it thrashing around trying to find a way forwards, or mired in meetings where you were trying to figure out the other guy’s agenda so you could get past another roadblock, or going down a path that was always destined to be a dead end had you only known it. Or trying to work out for the umpteenth time how stupid Java NIO sockets work. So it’s not really learning that’s the constraint – it’s ignorance. More accurately, it’s ignorance about specific aspects of the problem at hand. In other words:

Ignorance is the single greatest impediment to throughput.

Ignorance is multivariate

Ignorance applies along multiple axes. You can be ignorant of the particular technologies you are using, ignorant of the breadth of technology options available to you, ignorant of the domain, ignorant of the ways in which you could address the problem or opportunity, ignorant of a better way of articulating the problem – a better model – that would make the solution obvious, ignorant of the people in the team – their aspirations or fears, their motivation, their relationships with one another and out into to the wider organisation, ignorant of organisational constraints, ignorant of third party integration risks, ignorant of who are the people you should be building relationships with, ignorant of the delivery methodology, ignorant of the culture of the organisation. I’m just scratching the surface here – I’m sure you can imagine many other factors that could affect our ability to deliver and about which we will be more or less ignorant.

More insidiously, you are usually ignorant of how ignorant you are, and rather than making you more wary, this second-order ignorance actually makes you more likely to rush in like the proverbial fool. This is beautifully summed up in this quote from IRC snippets site, bash.org:

<Pahalial> “ignorance more frequently begets confidence than does knowledge” – Charles Darwin
<kionix> wtf? begets isn’t a word. quit trying to make up words [...expletive...]

Discovery is non-linear and disjoint

Now, think back to that project you just completed. Did your ignorance decrease consistently and linearly along all these axes? It’s fairly unlikely. What probably happened was that at various points – some more memorable than others – you had sudden insights or realisations, that either came to you or were thrust upon you by circumstance. Chances are that a lot of this unplanned learning happened fairly late in the day, accompanied by much disbelief, anger, and probably all the other stages of grieving too, as you come to terms with the death of your strongly-held model of How Things Ought To Be.

So for any single factor, you start the project with a particular level of ignorance, and it decreases in “bumps” – in a disjoint fashion – with each learning episode, until at the end of the project it is at another, lower level. Of course many of these factors are interrelated, so in a single episode you will typically learn about several different things – your ignorance of various factors will decrease by different amounts simultaneously. In reality much of this learning is at the whim of the Project Gods, and largely out of your control. How could you have possibly guessed that the third party API would be that different from the spec? Who knew Dave’s wife would choose that weekend to have the baby?

Well actually you could have. Ok, you couldn’t have known that those things would happen, but you would be insane – or rather “normal” – not to think something would happen, and this is the crux of Deliberate Discovery.

What if you assume something bad will happen?

We have a built-in mechanism for mindless optimism. It’s called attribution bias, and we use it to protect our fragile egos from the big bad reality out there. You can read more in Cordelia Fine’s fascinating book A Mind of Its Own, but in short, it means we assume when bad things happen to other people, they probably deserve it. They screwed up, or they didn’t plan ahead, or, well, any number of reasons. But when bad things happen to us, well that’s different. We couldn’t possibly have seen that coming! That could have happened to anyone. Poor us.

We are susceptible to attribution bias when we estimate (as Linda Rising has pointed out), and when we assess risk, and we are therefore constantly amazed – and let’s face it, a little disappointed – when bad things happen on our projects. Here’s another thought experiment: What if instead of hoping nothing bad will happen this time, you assumed the following as fact:

• Several (pick a number) Unpredictable Bad Things will happen during your project.
• You cannot know in advance what those Bad Things will be. That’s what Unpredictable means.
• The Bad Things will materially impact delivery. That’s what Bad means.

How would this affect your approach to the project?

Finally, deliberate discovery

So, I think we’ve been looking in the wrong place. Methodologies, practises and patterns for delivery are all well and good, but they don’t take into account the single biggest limiting factor to successful delivery. Let’s assume that during the life of the project our ignorance will reduce across a number of axes that are relevant to our project. Let’s also assume that ignorance of certain factors right now are the things currently limiting us the most. Let us further assume that we probably don’t know which ones those magic enabling factors are: we are second-order ignorant of which factors are currently the most constraining. Once we realise what these are, we can apply methodology to consistently move forwards, but until we do, we’re shooting in the dark.

Surely it makes sense to invest effort in firstly discovering which aspects of delivery we are most critically ignorant of (i.e. both where we are ignorant and where that ignorance is hampering throughput), and further to invest in reducing that ignorance – deliberately discovering enough to relieve the constraint and allow us to proceed. And all the way through the project, day by day, we should be trying to identify where and how ignorance is hampering us. Ideally we want to create as steep a descent as possible for each axis on the curve of ignorance, so we deliberately reduce the degree to which we are constrained by that ignorance, rather than being a victim of circumstance.

This is a skill, and as such is subject to the Dreyfus model. Which means that initially we’ll be rubbish at it. Then we’ll start to figure out how we are rubbish at it, and work on that. Then we’ll figure out some Best Pracises (I’m sorry – it’s inevitable. That’s what Competent people do to “protect” Advanced Beginners), and, hopefully shortly after, figure out ways to subvert those Best Practises to continue getting work done. The hardest part is going to be the damage to our egos as we realise just how systemically poor we are at delivering projects, and how we’ve been staring straight past the problem, because our methodology is always under the street lamp, and we feel safe under the street lamp.

What next?

This then comes back to my original premise last year, which is that during an inception, when we are most ignorant about most aspects of the project, the best use we can possibly make of the time available is to attempt to identify and reduce our ignorance across all the axes we can think of. (Arguably one of the first exercises should be to take a first stab at identifying these axes, and trying to figure out just how ignorant we are. How’s that for an exercise in humility!) Sure, if we stick to the traditional Agile planning model, we will do some discovery as we break down epics into features into stories into scenarios, but how much more could we accomplish if we put that to one side and instead focused on the real deal?

Domain-driven design inventor Eric Evans describes up-front analysis as “locking in our ignorance.” He’s got a point, and it doesn’t just apply to old-school requirements analysis. The death-by-stories planning exercises I’ve seen on many Agile projects bear testament to the same problem.

I hope this has given you an idea of where my head has been at. There is much more to say about deliberate discovery. Think about applying the principle to learning a new language, or picking up a new technology, or a new domain. What could you do to identify and reduce your ignorance most rapidly? Why are rapid-feedback teams so much more successful than long cycle teams? How can we measure ignorance? I’ll be writing more on this topic over the coming weeks and months, and I’ll be talking about Deliberate Discovery at QCon, San Francisco in November.

Thanks to Liz Keogh, Lindsay North, Joe Walnes, Chris Matts, Steve Hayes, Kevlin Henney and numerous others for helping to grow the ideas in this article. And special thanks to Glenn Vanderburg and Mike Nygard for the conversation at JAOO Australia about a unit of ignorance. (That will have to wait for a future article.)