Key Takeaways
- Humans vs Computers is a collection of stories, some amusing and some quite scary, about everyday people caught between software bugs and binary logic
- Modern software delivery is a constant struggle to abstract, simplify and model some part of the real world into a useful automated process. However, lack of domain knowledge, time pressure and imperfect information often lead us to oversimplify the real-world, so edge cases fall through the cracks
- Given how much software is now becoming part of everyday lives, things that seem like small bugs can have a serious impact on people
- Most software teams todays are so disconnected from end-users that they can't even imagine how something as trivial as a decimal format display bug can send people to a hospital.
- Hopefully, this book will also help readers find more empathy for people suffering from our mistakes, as it's always people who pay the price in the end
Software developer, author and thought leader Gojko Adzic has released another book. Adding to his work on Impact Mapping and Specification by Example, his latest book is Humans vs Computers. The book contains stories of the impact of software bugs and failures in the real world, and advice for software developers on how to avoid them.
The theme of the book can be summed up in this quote:
Our lives are increasingly tracked and monitored by software. In this brave new world, humans can't cope with information overload. Governments and companies rely on computers to automatically detect fraud, predict behaviour and enforce laws. Inflexible automatons, barely smarter than a fridge, now make life-changing decisions. Computers are determined to follow instructions to the letter – but the instructions are human, and flawed! The results can be unexpected, catastrophic, and very, very funny.
A sample extract of the book can be downloaded here
The printed version can be purchased here.
Gojko spoke to InfoQ about the background to the book, why he wrote it and who it's for.
InfoQ: What prompted you to write this book, what is the problem you're looking to address with it?
Humans vs Computers is a collection of stories, some amusing and some quite scary, about everyday people caught between software bugs and binary logic. From the man whose new personalised number plate brought him 2,000 parking fines a month to the prisoner released 3 years early by an automated parole review, these cautionary tales will hopefully help people prevent, avoid and reduce the impact of such problems in their own work.
Modern software delivery is a constant struggle to abstract, simplify and model some part of the real world into a useful automated process. However, lack of domain knowledge, time pressure and imperfect information often lead us to oversimplify the real-world, so edge cases fall through the cracks. For example, complex distributed systems built around microservices often require some kind of production monitoring that tries to process transactions end-to-end with test data, and remove those test cases at the end of a successful check. It's difficult to imagine how something like that can cause serious damage, until you know that someone called Jeff Sample ended up stranded in Buenos Aires when the airline operating the connecting flight deleted his ticket without any trace.
In retrospect, it's easy to see that someone made a silly oversight about what makes a valid name, but that doesn't really help a person who has to spend hours on the phone arguing with travel agents, card processing companies and airline representatives to be able to continue his journey.
Given how much software is now becoming part of everyday lives, things that seem like small bugs can have a serious impact on people. Most software teams today are so disconnected from end-users that they can't even imagine how something as trivial as a decimal format display bug can send people to a hospital.
I wanted to raise awareness about these issues, and help people build better software by preventing common blunders.
InfoQ: Who is the intended audience, what will they take from it?
The book is aimed at anyone involved in software delivery, regardless of their role. People involved in requirements and product management will get some nice ideas on how to avoid typical analysis blind-spots. People involved in architecture, design and development will learn about common over-simplifications and learn to ask better questions when discussing implementation details. People involved in testing and quality improvement will get some nice examples and heuristics to try out, and learn about the root causes of certain categories of problems, so they can be solved earlier in the process.
The categories of problems I wrote about are perfectly predictable and preventable, and there really is no excuse today to create those types of bugs any more. They've been already documented in various testing books, but tend to be listed as taxonomies of heuristics, difficult to remember and boring to read. And generally, they end up read by testers only, with very few developers caring to learn about them. And of course, under time pressure, it's difficult to recollect all those taxonomies instantly, so people end up introducing the same types of bugs over and over.
That's why I wanted to illustrate problems with memorable stories, in a book that will be interesting to read to anyone involved in software delivery, regardless of their role. Stories make ideas stick in our brains. You might not recollect all possible ways of messing up test data the next time you work on monitoring, but you'll remember Jeff Sample, and this will give you a jolt to think a bit harder and prevent similar issues.
Hopefully, this book will also help readers find more empathy for people suffering from our mistakes, as it’s always people who pay the price in the end. This alone should help to deliver better software, regardless of remembering any single specific technique.
InfoQ: You've split the book into a series of sections dealing with different types of software related problems – what are they and why did you use this structure?
The first part of the book, "Artificial, but not intelligence", deals with typical mistakes in decision making algorithms, caused by oversimplifying the real world. Some of the nice stories from that part explain how twins break biometric checks, how time-zone confusion can cause traffic chaos and how geo-IP mapping problems causes people to be wrongly accused of serious crimes.
The second part, "The surprising facts about our world", contains stories about strange and wonderful edge cases that break seemingly sensible validation rules, such as people with just a single name, or with too many characters to fit on a driver's license. Some of the nice stories in this part explain how an investment bank lost almost US$100 million in two minutes due to a silly decimal validation issue, and what kind of problems end up happening to people whose last name is Null.
The third part, "Algorithms as fast as food", covers typical oversights and problems in process automation. This includes stories of algorithms getting stuck in self-reinforcing loops and leading to stock market crashes, a t-shirt company imploding when its design software started advocating violence towards women, and the lavish lifestyle of an Australian man who got unlimited credit from his bank due to a software glitch.
The fourth part, "Wild, wild tech", covers problems that occur because of a problematic mapping between real-world concepts such as time, money and quantities to numbers in a computer. This includes stories about how a decimal rounding error changed the outcome of elections, how a 107-year old women had to fight against the local council trying to enroll her into school and how a discount coupon caused a wrestling match involving hundreds of people.
InfoQ: Which is your favorite story from the book and why?
If I had to choose one, it would probably be the story about Robert Barbour from Los Angeles, who wanted to get custom license plates for his car. His top two choices were BOATING and SAILING. Barbour didn't want anything else, but the computer form had three fields and they were all mandatory. So in the third choice field, he wrote 'no plate'. And, of course, the license plate was issued for his third choice. That looked quirky enough so Barbour decided not to complain. A month later, he started receiving notices for overdue parking fines from all over California. When an illegally parked vehicle did not have a license plate, the officers still had to issue a ticket, but computers just wouldn't take no for an answer. Officers had to find a workaround and, humans being humans, many had the same idea as Barbour. The officers just issued the fine for 'no plate'. Those tickets would normally not end up anywhere, but all of a sudden computers finally had something to match them against. By the time Barbour's trouble caught the attention of the media, he was getting more than 2000 tickets per month. He even had to appear in court a few times and explain the situation. Once the problem became political, because all the related systems could not be updated quickly, the authorities just issued a guideline to record cars without plates as 'MISSING'. But, of course, someone already had that exact license plate, so the cycle just repeated itself.
It of course easy to assign blame. UX designers created those inflexible forms, requirements analysts did not cover the case of cars parked without license plates, developers wrote queries that matched missing data with valid entries, and testers failed to discover this bug. Someone is always guilty in retrospect. But the more interesting question is why such a problem happened in the first place.
I love this story because it so nicely illustrates what happens when computers insist that people provide a piece of information that isn't immediately available or just doesn't exist, so users find workarounds. The real fun starts when several systems need to exchange information, and they all use different workarounds. The placeholders and markers in one part become valid data for another, and errors multiply and propagate. Problems with data markers can stay dormant for a long time. Once all the records that were never expected to match anything start to relate to a valid piece of information, such as someone finally getting the 'no plate' plates, chaos starts.
This whole situation is perfectly preventable if people involved in delivery just consider a bit more that making things mandatory does not necessarily lead to the right data being put in, quite the opposite. I often have trouble with my book printer that requires a phone number for delivering orders, and I rarely have that piece of information. So I just put in my phone number. But then I get calls in the middle of the night from lost couriers somewhere half-way around the world.
InfoQ: The last chapter deals with the "Inverse Monkey Rule" – what is this and what is the advice for technologists from this section?
My intention with this book is to illustrate some typical, common mistakes with memorable stories. However, rather than just having some funny stories to tell over drinks, I wanted to leave readers with clear actionable advice as well. The final part of the book contains some nice tips and tricks, combined from all the stories, with heuristics for analysing, developing and testing computer systems.
The name comes from Émile Borel's famous infinite monkey theorem, proposed in 1913. Borel suggested that given infinite time and attempts, monkeys would come up with the works of Shakespeare. Borel’s theorem is a nice illustration of statistics and calculus, but in practice the probability is infinitely small. On the other hand, inverting the subjects and the outcome gives us something a lot more practical, in much less time, causing most of the issues I covered in the book. "Smart people, hitting keys intentionally on a computer keyboard, given just a few months, will almost surely produce some kind of monkey crap." I named the principle the Inverse Monkey Rule.
The last part is by no means a comprehensive list of test cases or a full testing strategy, just a quick digest of the stories in this book. My intention is for people to use it as a check-list along with their other tools, and as an inspiration when looking for more ideas.
About the Book Author
Gojko Adzic is a partner at Neuri Consulting LLP. He is the winner of the 2016 European Software Testing Outstanding Achievement Award, and the 2011 Most Influential Agile Testing Professional Award. Gojko's book Specification by Example won the Jolt Award for the best book of 2012, and his blog won the UK Agile Award for the best online publication in 2010. Gojko is a frequent keynote speaker at leading software development conferences and one of the authors of MindMup and Claudia.js. As a consultant, Gojko has helped companies around the world improve their software delivery, from some of the largest financial institutions to small innovative startups. Gojko specialises in are agile and lean quality improvement, in particular impact mapping, agile testing, specification by example and behaviour driven development.