Examples of bad software – how much damage can poor code do?
Real-life cases in which disastrous software failures led to enormous losses and even took a human life
Everybody knows bad software is out there. In our imperfect world, a handful of coincidences – human mistakes, buggy code, or unexpected circumstances, for instance – can bring down even fairly solid systems. Let’s look at real-life cases in which disastrous software failures or errors led to enormous losses and even took a human life.
A software bug at the UK Post Office got 736 innocent employees convicted
For 20 years, the UK Post Office relied on software called Horizon. Because of bugs, it reported money missing from accounts that employees were responsible for. It appeared as if staff had stolen thousands. In the end, 736 post office operators were convicted. People lost their jobs and families, and a pregnant woman ended up in prison. One man took his own life after the system claimed £100,000 was missing from his account.
The case is all the more controversial because evidence suggests the legal department was aware of the system’s problems before the convictions. The Post Office has begun offering compensation and says it will swap the Horizon system for a cloud-based solution.
TUI airline got its flight loads wrong
In 2020, the loads of three flights were calculated incorrectly. TUI’s check-in software classified passengers with the title “Miss” as children. Since passenger weight is used to work out the thrust needed for take off, this resulted in an unlucky miscalculation. A child counts as 35kg and an adult as 69kg. A lower computed weight means less thrust at take off. With an unlucky passenger list, this could end in a disaster. Luckily, the final thrust value stayed within the safety margin, and all passengers reached their destinations without trouble.
A UX flaw cost Citibank $500 million
Source: court document
Ever heard of Oracle FLEXCUBE? It’s a banking system that Citibank uses. In 2020, staff meant to send roughly $7.8 million in interest payments. Because too few fields in the form were filled in, nearly $900 million went out. What’s interesting is that a transaction this large has to be approved by 3 people, and in reality all three believed the form was filled in properly. Without going into the legal details, Citibank ended up never recovering about $500 million.
The false missile alarm in Hawaii
In 2018, the emergency alert systems in Hawaii sent out warnings about incoming ballistic missiles. This triggered mass panic: some people hid their kids in sewers, while others recorded farewell messages for their families. The mobile network was completely overloaded, and people couldn’t get through to 911. Sending a message that there was no threat and canceling the alarm took 38 minutes.
The incident was analyzed in detail, and several problems were found among its causes, including a poor UI and errors in communication between people. The employee who triggered the alarm lost their job. The alarm procedure was reworked, and starting an alarm now needs confirmation from 2 people.
Uber was sued for $45 million over a notification shown after logging out
The Uber app had a bug that made it display notifications even after the user had logged out. Dangerous? Not quite. But in reality, a French businessman was having an affair, and notifications about his rides went to his wife’s phone. How come? He had once used Uber on her phone and then logged out. The bug affected only the iPhone version and has since been fixed. The couple got divorced, and the Frenchman sued Uber for $45 million.
Revolut lost $20 million
At the beginning of 2022, more than $20 million was stolen from Revolut. It turned out that, because U.S. and European systems differed, some declined transactions were still refunded with Revolut’s own money. The refunded sums were then taken out of ATMs. The bug had likely been there since 2021 and was fixed in spring 2022, after a Revolut partner reported that company money was missing. Various malicious actors took advantage of the vulnerability, stealing more than $20 million this way.
A Nest Thermostat update left users freezing because of software bugs
Do you have a smart home? Google makes the Nest smart thermostat. In the winter of around 2016, a software fault drained its battery and, as a result, switched the heating off. A winter with no heating? That can create plenty of trouble, and even more for some people, as certain users were away and had set the thermostat to keep their pipes from freezing.
This wasn’t the only fault in Nest software over the years. If you use IoT or Smart home devices, remember that updates or infrastructure outages can affect what works in your home.
Knight Capital Group lost $440M on bad trades
Knight Capital Group used automated trading software. Because of several bugs and mistakes made by human operators, the system bought hundreds of millions of shares within 45 minutes. It turned out that a new code release hadn’t been deployed to one of the company’s servers, while that release also reused an old flag with a different meaning. The flag was switched on across all servers, running both new and old code, which caused old, unused test functions to run and generate all those orders.
These operations cost the company $440 million, and its share price crashed. As a result, a competitor acquired it within the following year.
The huge Equifax data breach
This is one of the biggest stories of last year. Equifax got hacked, and the attackers got hold of data belonging to hundreds of millions of people. How did it happen? Once more, for several reasons. The systems hadn’t been patched against a known vulnerability, even though the administrators had been warned.
On top of that, many other poor security practices came to light, such as insufficient separation of internal systems and passwords kept in the system as plain text. The hackers had access to the data for months before anyone noticed. Following the incident, Equifax spent $1.4 billion on improving security.
Toyota software glitches cost 89 lives
Toyota was forced to recall over 8 million cars because of software errors. Some cars sped up even when nobody pressed the gas pedal. The investigation revealed that the systems were poorly designed and of low quality and contained numerous software bugs, such as memory corruption, buffer overflow, unsafe casting, race conditions, and more. In practice, the whole affair dragged on for years. At first, Toyota blamed the problem on floor mats. The company was fined $1.2 Billion for hiding safety defects. The most critical piece of code responsible for acceleration turned out to have enormous cyclomatic complexity, which effectively made it impossible to test.
Conclusions
Stories like these are plentiful, and we could keep listing major software failures for a long time. What lessons do they teach us? Software surrounds us. It’s present in many areas of our lives – our homes, cars, healthcare, and work. Poor quality and bugs can ruin lives, kill people, or lead to massive financial losses. This makes it obvious how much a responsible software team matters, how much security and software quality attributes matter, and how much the UI and the UX matter!
Any carelessness, such as skipping updates of vulnerable libraries, web servers, or operating systems, can, combined with other factors, result in huge data breaches. Today, the software development process should include a range of procedures and practices that help prevent such tragic situations. How? It should cover, among other things, security audits of computer systems, UX tests, and adequate test code coverage. We must keep in mind, though, that even with all of this in place, people still make mistakes. As the examples show, the worst software failures come from several different factors overlapping. A single human decision shouldn’t lead to a problem, but that holds only when the entire development and operation process is sound.
You can’t remove every risk from your business, but a skilled software development team that communicates well is essential for getting closer to that goal. Building software is hard, and you need a proactive technology partner who will not only build your software projects but, above all, deliver innovation with real value, so your business can grow without expensive mistakes.