A bad (computer) system will beat a good person every time

This is a story about amazing customer service being undermined by poor software, in the form of simplistic business rules and fragile systems. I am telling it because I made a promise to someone who features prominently in the story, and whose manager should be aware just how fantastic their staff are, and just how much their poorly designed computer systems and cost-cutting are letting their company down.

W. Edwards Deming famously said “A bad system will beat a good person every time.” He was talking about systems of human interaction but his words apply equally to computer systems.

A recent trip to South Africa got off to a truly bad start. We had wanted to spend three weeks there but due to a work scheduling conflict we could only go for two weeks, so every day would be precious. My wife would visit friends and family she hadn’t seen in years; our four-year-old son would be making new memories—he was an infant the last time they travelled there.

We were flying with British Airways. I have been a loyal customer for years and before the pandemic I had gained their top-level status, having spent most of previous two years commuting to India from the UK every month, which is a great way to rack up points. They had generously extended everyone’s status expiry by a year and we had booked business-class tickets using an insane amount of Avios loyalty points.

Part 1, in which “our systems are down” ¶

We arrived at London Heathrow’s Terminal 5 on Wednesday evening to check in early for our overnight BA flight to Cape Town. The airport was crowded and in turmoil. BA had experienced an unspecified technical issue earlier in the day which meant that the desk staff were checking everyone in manually, and they were cancelling and rescheduling flights to clear the backlog as a result. We had not seen the news reports so we arrived unprepared for the chaos.

We went to the First Class check-in area—one of the perks of status—and joined the unexpected and depressing queue to find out whether our flight was even leaving. Eventually we got to the desk to be told that our 10:30pm flight was now delayed to 11:30am the following day.

The lady tried to check us in, only to discover that we could not do this yet. We were given a voucher for a nearby airport hotel, which was exactly what you would imagine, and told to come back in the morning.

Lesson: It is 2022. Observability is a thing. Resilience is a thing. Intraday outages of critical business systems should no longer be a thing.

The enormous queues for the tiny shuttle buses were nobody’s fault, and after waiting in line for half an hour or so with a four-year-old we paid a taxi £20 to take us across the airport to the hotel.

Part 2, in which there is no spoon ¶

We arrived back at the check-in desk at 8am the next morning, partly because we did not get much sleep, partly to get ahead of the queues. This was when we met Lily (all names have been changed). We handed over our passports and Lily spent some time tapping things into the terminal, looking puzzled, tapping in some more things, looking more puzzled, and eventually asking us: “Do you have tickets for this flight?”

Why yes, we declared, we reserved our seats, booked special meals, and here we are! But do you have tickets for this flight?

Wait, what?

Well, you see, there is a difference between a “booking” and a “ticket”. You can have a booking—a seat, a meal, a six-character locator code, all the things that you would expect—without a ticket ever being issued!

What had happened was something like this. My assistant booked the tickets using Avios points—British Airways’ loyalty scheme—leaving only the airport taxes to pay on my credit card. The BA agent took the credit card details and said that the payments team would complete the booking, and that “Avios bookings usually take 24 hours to go through.” My assistant checked the following day and saw a booking in my account, with a locator code, seats, all the usual stuff. As did I, which is when I chose our seats and booked special meals for wife and son.

But, crucially, there were no tickets. There was no indication that there were no tickets. There was no suggestion that the booking was anything other than booked (which, technically, it was; it just wasn’t ticketed!).

Lesson: If your business rule sounds stupid, it probably is. Change it.

So here we were, and now we understood why they had been unable to check us in the previous evening. It wasn’t because the flight was delayed, it was because no tickets had been issued.

“So,” I asked, “if we don’t have tickets but there are three seats reserved on the flight, what happens to those seats?” “The plane will fly with three empty seats.” Ok, good to know. So, can we get tickets issued then?

Lily sprang into action. “You will get on that flight!” she declared, and immediately phoned… the same number anyone else phones to check a booking. In a cost-cutting move, BA had decided that their desk staff did not need special access to the booking operations team and that they should use the same number as anyone sitting at home. They had also got rid of the two ticketing specialists they used to have alongside the check-in desks to save another few bucks.

Lily was stuck in the same automated system as everyone else, being assured that her call was important to BA, that she was held in a queue, and that someone would be with her shortly. She was calling from her personal cellphone because BA had also decided that their desk staff should not be allowed to use their desk phones as actual phones. An hour or so later, with her phone battery running dangerously low, she asked how and when we had paid for the tickets (while running around looking for a plug socket).

When I mentioned the word “Avios” the colour drained from Lily’s face. “That’s a different number. The BA Avios team is separate from the regular booking team. I’ll have to redial.”

Lesson: Organise by customer need, not by internal accounting code. Issuing a ticket should be independent of how a customer pays for their fare.

While she was on hold for the second time, to a different team, Lily asked me to check my credit card; sure enough there had been no transaction. The payments team had not taken the payment, had not issued the ticket, and no one had noticed because tickets. Lily was beside herself. “This is definitely a mistake at our end, and I will get you on that flight.”

Part 3, in which we encounter a quantum aeroplane ¶

After another 45 minutes on hold (“Did you know you can manage your booking on our website?”), and telling lies to the automated voice prompt so it didn’t hang up on her, Lily eventually got through to a human being, called Ellie.

Lily explained the situation to Ellie, who quickly realised the urgency of the situation. It was after 10am by this point and the flight was due to leave at 11:30. Airline security rules mean that no tickets can be issued within an hour of a flight, so there was a hard cut-off in less than 30 minutes.

Ellie jumped onto her MS Teams chat, keeping us updated in real time, liaising with the ticketing operations team, determined to get us onto the flight. It seemed that the key constraint to issuing our tickets was Ellie’s typing speed and it looked like Ellie was winning!

Suddenly things went quiet. A strange system error was preventing Ellie from issuing the tickets. It turns out that BA’s systems are unable to issue a ticket for a flight that has already left, which makes perfect sense. Our flight had not yet left, of course. The plane was there, in the terminal, starting to board passengers. (Lily had her mobile phone to one ear and her desk phone to the other, speaking to the boarding desk to hold the flight!)

Ah, said the ticketing system, but it should have left last night, and that’s what matters! But, said reality, it clearly hasn’t. It leaves in just over an hour. It is sitting right there. Can we just get the… But it should have, shouldn’t it? And because it should have, I am unable to issue your tickets.

Lesson: Think about your business rules, especially time-based rules. Consider edge cases. Study real scenarios.

So here we were. Lily was doing everything humanly possible; Ellie was doing everything humanly possible; our plane was in a quantum superposition of being both on the tarmac this morning and having taken off last night, as long as you didn’t look at it.

Lily was distraught; Ellie was apoplectic (go Ellie!); my assistant was in turns incredulous, apologetic, and furious; there had already been tears. And we were most definitely not getting on this flight.

The zero hour of 10:30am came and went. The plane duly took off at 11:30 without us. Lily and Ellie, who was still on the line, were determined to make things right. Ellie was pulling whatever strings she could with the operations team to get us to Cape Town as quickly as possible. The Thursday flight was full so the earliest they could get us there on was on an indirect flight via Johannesburg the following night, connecting on Saturday lunchtime, with a layover of several hours in the Johannesburg domestic terminal.

Ellie took my credit card details and issued the tickets herself. There was nothing more to do now until tomorrow night. Meanwhile, Lily had exchanged phone numbers with my wife and they were making plans to meet up for a drink.

By now we had lost the Thursday and Friday, and the Saturday was not looking good. The connecting flight would get us to Cape Town late in the afternon.

Part 4, in which the flight does not exist; the flight has never existed ¶

We decided to go home for the remainder of Thursday and come back again Friday afternoon. Neighbours were surprised to see us. Son was stoic. “Will we get on a plane next time, Daddy?”

I figured I would at least try to get some work done on the Friday, and decided to start my day by checking us in on the flight—for which we now definitely had tickets—so I could print the boarding passes like a Luddite obsessive. I logged into my account and saw… my next flight was for some time in October, to somewhere else. The Cape Town flight had vanished.

Convinced that I was going mad, I logged out and went to British Airways’ “Manage my booking” page instead, and typed in the locator code. “There are no tickets against this locator code.”

Blink.

Breathe. Just breathe. And for heaven’s sake don’t tell wife. Not yet. Ok, let’s call BA’s special number for gold card holders. “Please select from the following options.” Tappety-tap tap. Wrong choice; the call hangs up. This is like a choose-your-own-adventure phone call. Game over, you are eaten by a grue. Ok, time to lie. Why yes, I would love to book a flight. Yes it is leaving today. “You are in a queue.” Phew, at least now, after holding for an hour, I might get to speak to… “Hello, this is Mike.”

So I explain to Mike the events of the past two days, including the booked-but-not-ticketed flight, the rebooked flight, and now the disappeared flight. Mike taps around for a bit. Mike has figured it out. It seems that because we never boarded the original flight—the quantum one for which tickets were never issued and for which we were unable to get tickets on the day, but on which we had seats booked, that flight—because we never boarded it, we count as a “no-show”. This immediately cancels all remaining legs of the journey, which in our case was all of them. Mike is sorry. Mike agrees that this makes no sense. Mike is determined to resolve this. Mike puts me on hold. It is a very long, very quiet 20 minutes.

Mike has spoken with his supervisor and they have managed to reinstate the flights. I breathe out for the first time in forever. I learn that it is possible to apply “no-show protection” to a ticket to avoid this happening, which must have been overlooked in all the excitement of the rebooking.

Lesson: When a business rule has serious consequences, think through the implications of those consequences. How could you mitigate them?

I am assured I will receive an e-ticket email in the next few minutes. “Would you mind staying on the line until the email comes through, only I’ve had a pretty rotten couple of days?” Not at all, says Mike.

Twenty minutes later, the email has not arrived. Other emails have arrived, so my email is working ok. Mike assures me—tappety-tap—that their email systems are working, and more importantly that the tickets have definitely been issued.

Part 5, in which we cannot choose our seats because reasons ¶

Things are looking up. I try checking in online because surely we are all set now.

“You are unable to check in because we have not received your COVID vaccination details.” Wait, I already did this! Using their app that I had to install and upload documents into. Against a different flight code. Which doesn’t count, because that flight already left.

I hastily re-download, re-upload, re-validate documents, passport details, photos for the three of us. “We will check these details as quickly as possible.”

At least I can choose our seats; hopefully we can all sit together, right? Wrong. “You cannot choose your seats at this time because check-in is open. Please choose your seats during check-in.”

Brilliant, all the options for changing the booking are disabled once check-in opens and become part of the check-in user journey. Meanwhile: “You cannot check in at this time because we are validating your COVID documents.” I don’t want to check in, I want to choose seats. Before everyone else checks in and chooses theirs. Because one of us is four years old. Do not hire George Orwell as your business analyst.

Lesson: Do not close off one user journey just because another becomes available. Allow multiple paths, support each of them.

Meanwhile Mike—remember him?—is stil on the line. The email still has not arrived. I thank Mike and let him go. We are not checked in. I do not have tickets. It is late morning on Friday. I try to concentrate on work but I am distracted, checking the BA page every twenty minutes or so, until… Hooray! Documents are checked and approved, I can check in, and I choose seats that are at least within line-of-sight of each other.

Part 6, in which we finally get on a plane and only have one more hiccup ¶

Friday afternoon we were back at the airport, back at the same check-in desk, meeting our third check-in agent. We asked about sitting closer together—son was separated from mom by a divider and from dad by an aisle. Moving us was “not possible” so we would have to figure it out. Still, the agent was checking our bags—checking our bags!—so we were going to fly. We went through to the security gate, the air side of the terminal finally in sight.

“There seems to be a problem with your boarding pass. You need to go back to the desk to get it sorted out.” Of course there was a problem. Why would there not be a problem? Can you tell me what it is? No, you will have to go back and talk to the check-in desk.

So back we went to the desk, to another agent—our one was dealing with another customer now—who told us that the bags had been checked in against the wrong passengers. Do you mean different people? Oh no, definitely us, but get this. When I checked us in online, the form asked how many bags we were checking, and I had randomly allocated two, one, and one against our names. When we checked the bags at the desk, the agent had, equally randomly, allocated the bags differently. Same four bags, same three passengers, different boarding passes scanned. Which apparently raises a security alert that requires returning to the check-in desk.

The agent at the check-in desk assured us that the person scanning our boarding pass could have resolved it there and then. Or rather the agent standing next to her said this. The agent we spoke to said that this could not be resolved at the desk and that she needed to call someone. The agent next door must have seen the incredulous look on my face, because she asked her customer to wait a moment and staged an intervention.

Lesson: If a business rule depends on the individual who is applying it, you have a training problem.

Part 7, in which we spend the entire day in an airport ¶

Finally we were on a plane, taking off, hoping that our luggage had made it onto the same flight. We arrived the next morning in Johannesburg at 10am to discover that our 2pm flight to Cape Town was now delayed to 5:30pm. Our seats on the flight had our four-year-old son sitting next to a complete stranger a couple of rows behind us. We were in an exit row, for which you need to be over 18 years of age, so neither of us could swap with our son.

Fortunately, Carl, the check-in agent at Johannesburg was fantastic. He allowed us to check our bags early and moved a couple of seats around so that our son was now sitting with a parent. We spent a frustrated but pleasant enough day in the airport terminal. There was a food court with a soft play area and we got our “travel admin” done, which mostly involved picking up local phone SIMs.

Lesson: If you give people the autonomy and the tools to do a good job, they will often surprise you and delight your customers.

Had we known how delayed our flight would be, we could have made more of the day, or even tried to get on an earlier flight. Our 2pm flight finally left at 7pm and we arrived without incident into Cape Town at 9pm, picked up the hire car, and got to the accommodation around 11pm.

Epilogue, in which lessons probably will not be learned ¶

Due to a litany of avoidable failures, we lost three full days of a 13-day break. We still made the most of our time there and had some great adventures, several braais, and hung out with penguins on the beach.

We tried unsuccessfully to extend the trip. The AirBnB hosts were wonderful and offered us the accommodation over the Easter weekend but we were unable to rebook flights.

The return trip, our original direct flight from Cape Town to London, was blissfully uneventful. We had seats alongside each other, had impeccable service, even landed early.

I will continue to fly BA. They have amazing staff, on the ground, in the air, hidden away in their operations centres. I hope that someone there with enough authority reads this, sees the lessons, and decides that they care enough to do something about them.

As a technology and business consultant, I can understand how these systems are designed, how they evolve, how staffing and operational corners get cut, and how my situation involved the kinds of edge case that “never happen”. Until they do.

So this is for you, Lily and Ellie, Mike and Carl. Thank you for your dedication and tenacity in the face of technological mishap and farce. I appreciate you.