The most dangerous laptop on the planet was recently put on the auction. The old 10-inch Samsung on Windows XP has got a nice name — “The Persistence of Chaos,” and six malware programs installed on it: ILOVEYOU, MyDoom, SoBig, WannaCry, DarkTequila, and BlackEnergy, which altogether caused the damage for $95 billion. The laptop is isolated from the internet and other gadgets, so it is not going to cause any harm now.
Computer viruses, however, aren’t always the issue that causes problems. Lack of communication between teams, lack of attention, and poor software testing can have more horrific outcomes — from causing inconvenience for thousands of people to shocking financial losses or even people’s deaths.
So how much does a bug cost? It depends on the circumstances, and you can never predict the software development cost of bugs until you detect these weaknesses. We’ve decided to tell you about some headline-making cases that prove the importance of good QA unit and responsible testing.
What is bug in software testing (with example)
Before we move to cost of software bugs, a short reminder for the newbies. A bug in software testing is a program error or a functional defect in the software, which causes unpredictable behavior of a program or system. The goal of a QA team is to find all those defects, to detect the potential risks, and to pass on the findings to the dev team. Developers are to fix bugs before deployment and deliver the software of proper quality to an end-user.
You might have noticed some common bugs, too: incorrect localization features, troubles with supporting certain device models and OS versions, heavy power consumption, problems with connectivity and responsive design issues, unclickable buttons, etc. But these are just some minor issues compared to what you’ll discover below.
Houston, we have a problem: troublesome space investigation
If you are curious about the sphere with the highest cost of software defects, this is certainly space exploration. Elon Musk may have successfully sent his Tesla Roadster to the other planet on the Falcon Heavy, but many more launches that meant to be spectacular turned into disasters.
Mars Climate Orbiter
Mars Climate Orbiter
Cost: over $193 million
The robotic space probe was sent to explore the Martian climate and take photos of the planet on December 11, 1998. The way to Mars took around 9 months. On September 23, 1999, the Orbiter was supposed to produce a retroburn, head over to Highly Elliptical Orbit with a 4-hour period, and enter an orbit in two months.
At the designated time at 193-km height, Orbiter started to spin down. In five minutes, it headed towards Mars and disappeared forever. The analysis of mistakes showed that the probe lowered to 57 km over the surface instead of the planned 110 km and collapsed in the atmosphere.
The breakdown happened due to incorrect calculations. The teams that were working on the development used different measurements: NASA calculated the final stage parameters in the metric system, while Lockheed Martin engineers used the British system. As a result, the craft descended 53 km lower than it was supposed. $193.1 million were spent on spacecraft development alone, not including the price of the launch and mission operation.
Mariner 1
Cost: $18.5 million
The spacecraft Mariner 1 was launched in 1962 for a first-ever planetary flyby of Venus. Just in 293 seconds after the launch, the spacecraft went off the designated route. To prevent the damage for inhabited localities that could happen in case of a crush, Mariner 1 was destroyed high in the sky in 4 minutes 53 seconds after the start of the mission.
According to the official version, a software developer made a mistake when translating a handwritten formula into code: he mistook an index character for a dash. As a result, an onboard computer assumed normal speed steps for critical causing a breakdown. The spacecraft lost connection with controls on the Earth, and the system proceeded to Plan B. The New York Times called it the most expensive dash in history. Mariner 2 was launched two months later, with all the indexes in proper places.
Ariane 5
Cost: $8 million
Ariane 5 didn’t last that long: the heavy launch vehicle designed by the European space agency was destroyed on the 39th second of the flight due to incorrect software functioning.
The software installed on Ariane 5 (it’s inertial navigation, in particular) was originally developed for Ariane 4. Because of the different launch trajectory, the horizontal speed exceeded the set limits and caused a breakdown. Ariane 5 had a more powerful engine, which caused a bug that wasn’t typical for previous versions. The program attempted to record a 64-bit number into 16-bit space. Both main and backup computer failed because they were running the same programs. On the 39th second, the rocket started to collapse and self-destructed, induced by aerodynamic forces.
$8 million were spent on designing the rocket. The compound damage, however, was much bigger. The total price of satellites Ariane 5 was supposed to place into orbit amounted between $165M and $220M.
Something’s wrong with my computer!
Bugs in user software are much more common. In the end, people don’t try to launch rockets every day, while almost everyone has got several devices connected to the internet. Sometimes users have to deal with serious system errors, viruses, and gaming features that don’t function the way they are supposed to.
Pentium processor bug
Cost: $475 million
In 1994, Thomas R. Nicely, a math professor Professor at Lynchburg College, detected a bug in a popular Pentium processor and published an article about it. It was a Pentium FDIV flaw — a bug in operations module with floating decimal point in original Pentium processors produced in 1993. Due to the error, the processor might return incorrect binary floating-point results when dividing a number.
According to the official statement, it happened because of the missing entries in the lookup table used by the floating-point division circuitry. The Intel486DX version of 1989 used a coprocessor module with a floating-point. The microchip of the next generation worked with integers but the coprocessor remained the same. The probability of such a mistake was 1 in 9 billion so Intel announced that they were ready to replace processors in case a user proves the bug affected their work. Disgruntled customers, however, decided that every user deserved to get properly working hardware and requested replacements.
Morris Worm
Cost: $100 million
In December 1988, a graduate student at Cornell University Robert Tappan Morris accidentally created a malware program. It started as a harmless experiment but the program went out of control due to a mistake in a code. It started spreading rapidly, blowing out thousands of computers. Morris was accused of cybercrime and fined for $10,000, although around $100 million were spent for compensating the damage he caused.
Morris’s lawyer claimed that the worm helped to improve cybersecurity. Later, Morris became a co-founder of Y Combinator. He is an associate professor at MIT. A floppy disc with a worm’s source code is stored at Boston University.
WoW virus
Cost: virtual epidemic, price is not announced
This article wouldn’t be complete without mentioning some bugs in games, in particular, the case of World of Warcraft with its “bloody virus.” It started in 2005 when WoW developers introduced Hakkar the Soulflayer, the Blood God. His mission was to poison the enemies with corrupted blood during attacks. This cool feature turned into a real tragedy for players. The characters weren’t the only ones to be poisoned, but their virtual pets suffered as well. The blood turned into a virus and affected the characters for a long time.
As a result, experienced users somehow managed to survive, but the new ones were dying almost instantly. When the bug was fixed, Blizzard hurried up to announce the victory over the virus of corrupted blood. The scale of the damage for the company and players, however, was never announced, neither was the reason for the accident.
Windows authenticity questioned
Cost: thousands of users received notifications about using pirated software
In 2007, Windows XP users were informed that they were using unauthorized software. Windows Vista users have got more serious problems: some functions, including Aero and virtual RAM drives, were turned off.
The reason was ridiculous: the developers released the system that hasn’t been tested in a production environment. They noticed this software bug in half an hour, but the quick update release fixed only the issue with activations. The validation errors continued for 19 more hours, causing a lot of troubles for users.
Heartbleed virus
Cost: worldwide data leakage
In 2014, the IT community started discussing a software bug named “Heartbleed” that appeared in the OpenSSL library of TLS protocol. It was caused by the RAM stack overflow, letting any person connected to the Internet read user information up to 64KB from a computer running OpenSSL process. According to the reports, around 17% of all the protected websites worldwide were using this protocol and had this security-sensitive area.
Interestingly, Heartbleed was first mentioned two years earlier, but only Windows and Mozilla started working on the problem. Google decided to solve it by a quick patch. Thus, Heartbleed may be still traveling on the web and scammers may be stealing some private data right now for their new scam schemes.
Financial loses, nervous customers, endangered businesses
Big corporations and state bodies become victims of functional defects in software testing even more often. Again, there are many mistakes a good QA team could have prevented.
Child Support Agency messes up payments
Cost: $1 billion
In 2004, EDS developed a complicated computer system for payment of a benefit for the British Child Support Agency. Meanwhile, the Department for Work and Pensions resolved to reorganize the institution. Running these processes simultaneously induced serious errors: the system paid over the odds for 1.9 million people and was short in payments for 700,000 people.
$7 billion dollars didn’t make their way to accounts in time. DWP had to deal with 239,000 old and over 36,000 pending cases that stuck in the system. It cost over 539 million GBP (around one billion US dollars) to handle the situation.
Knight almost goes bankrupt
Cost: $440 million
Knight is one of the key stakeholders of the American stock market, and they almost went bankrupt due to a single software error. Because of the bug, the stock quotations declined by 75%, and the company lost around $440 million. The company settled for amalgamation.
The reason for the collapse was an error in the trade algorithm. During two days, the software flood the market with unplanned deals at approximately 150 trading sites shutting down the work. The software processed false deals, buying at too high prices, and selling at too low prices — doesn’t sound like a successful trade strategy.
In-demand British passports
Cost: $16 million
In 1999, a newly implemented by the British passport agency Siemens system couldn’t handle the timely issuing of passports for 500,000 British citizens. The company compensated for delays and for overtime. They even paid for the umbrellas for people in the lines.
It happened because the system hasn’t been tested and the staff wasn’t educated. The implementation coincided with the issuing of a new law, binding all children younger than 16 going abroad should have passports of a new standard. The extraordinary demand became a challenge the new system couldn’t handle.
Airbus A380 gets outdated software
Cost: a one-year delay of the release and the amount of money Airbus prefers not to talk about
In 2006, German and French divisions of Airbus started the development of two parts of the new А380. Both used CATIA software. When it came to assembling, engineers were surprised to find out that some cables couldn’t be connected — there was simply no place to connect them.
The miscommunication (or rather the lack of communication) resulted in the usage of different software versions, which required different hardware. The release was delayed for over a year. French division of the Airbus prefers not to talk about the financial losses.
Flight delays in Heathrow
Cost: 42,000 lost luggage items, 500 canceled flights
Shortly before opening the fifth terminal in Heathrow airport, the staff started testing the newest luggage transporting system. It was supposed to facilitate managing a large amount of luggage received at the airport every day. Before the opening, it was well checked on 12,000 test luggage items. Everything passed smoothly. In the first day of opening, however, it turned out that the system is unfit for service.
The program couldn’t handle some simple real scenarios which for some reason weren’t tested. For example, a person forgot something important in a bag, and the luggage was carried back manually. In this case, the program went off and an item wasn’t recorded. Luggage processing was disrupted by every little detail. During the following ten days, around 42,000 items weren’t delivered to owners, and over 500 flights were canceled. Check-in to other flights became temporarily unavailable.
A mess for insurance companies
Cost: 8,500 people declared dead (but luckily, they weren’t)
An extraordinary case happened in 2003 in a hospital in Grand Rapids, Michigan. A software bug in the patient database declared 8,500 people dead. All of them were safe and sound, but it was detected only after the investigation by insurance companies. The thing is, after a patient dies and a corresponding record appears in the hospital’s database, an insurance company receives a notification with the requirement to cover the treatment costs and pay compensation for the family.
Many people were surprised when insurance agents appeared on the doorstep. Nevertheless, there were no lawsuits or official compensations. Maybe some patients decided to change the hospital. The breakdown happened because of a mapping error after patient management software updates: a disposition code “20” was assigned instead of “01” — “dead” instead of “discharged”.
The headache for Washington state authorities
Cost: 3,200 criminals released early (without court decision)
At the end of December 2015, the authorities of the Washington state noticed a bug in a system that has been functioning for around 12 years. The state authorities decided to operate early releases based on good behaviour credits automatically. Prison sentences consist of the regular part and enhancement (added for crimes committed under certain circumstances). The so-called “good time” credits were to be applied to the regular part only, but an error made the algorithm work differently.
The analysis showed that in average prisoners got out 49 days earlier, although there was a case of cutting the sentence for 600 days. It turned out that the bug has already been reported in 2013 by a family of a victim — they found out that an offender was released too early. The police started to return people to jails.
Reputation seriously damaged
Cost: scientists are to deal with the poisoned reputation
Ironically, software for studying accidents became an accident itself. New England Journal of Medicine published an investigation about the increase in the number of suicides following natural disasters. The results turned out to be untrue. The software doubled the number of suicides for one year in the investigated period — quite enough to cause erroneous results. Although this case wasn’t followed by financial losses, their reputation suffered severely.
Fatal bugs: it’s not always about the money
Sometimes the cost of defects in software testing cannot be measured in money. A bug in medical or military program leads to numerous deaths. Unfortunately, these cases are numerous, too.
The USA launches five ballistic missiles, but do they really?
Cost: seconds away from starting a nuclear war
This is a story about the man who saved the world… by doing nothing. A bug in the Soviet’s Early Reaction System displayed a false report about the launch of five ballistic missiles from the US territory. The duty officer Stanislav Petrov, however, considered it a mistake and decided not to take further actions. He thought that if the US forces decided to attack, they would launch more missiles. The accident was included in a report as a false alert. But what happened then? A software error made the system mistake gleams at the top of the clouds for a ballistic rocket launch.
Patriot missile defense complex
Cost: 28 killed and 100 wounded soldiers
In February 1991, an American missile defense complex installed in Saudi Arabia missed a missile strike at barracks. It happened due to the software failure that resulted in tracking inaccuracy, which aggravated due to the long-term system exploitation. By the time of the blow-up, Patriot has been continuously working for over 100 hours. The defense complex predicted the trajectory of an attacking missile, but it was a roundoff error that calculated the time of flight incorrectly, and the defense mechanism wasn’t activated timely.
Before the incident, American engineers found this bug that caused inaccurate functioning — every 100 hours of continuous work of the complex resulted in a ⅓ second shift in calculations — and fixed it. The updated version of Patriot software was delivered the next day after the attack.
Therac-25
Cost: 4 patients died, 2 received an overdose of radiation
In 1982, Canadian company AECL presented a computer-controlled radiation therapy machine Therac-25. The difficulty of testing a part of the software code responsible for estimating a patient’s condition led to a failure that resulted in giving massive overdoses of radiation. Six cases were registered between 1985 and 1987, four of them lethal.
The machine had two work modes — direct therapy and megavolt X-ray therapy. In the first case, the electron beam was directed at a patient directly, when the radiation dose wasn’t very high. In the second case, the beam hit a metallic target that distributed radiation beams and then directed at a patient.
Earlier models of the device were equipped with physical detectors of the target presence to make sure the beams aren’t directed at a patient when they are going to receive a large dose of radiation. In Therac-25, physical detectors were replaced by program ones. Due to an arithmetic overload, the system started using in calculations a number that was too high for operations. If it happened during the setup, security systems failed and the target wasn’t placed where it should be, and a patient received 100 times higher radiation than was supposed. QA team found four other critical bugs, which caused synchronization failures and defined wrong setup parameters.
Software failure in Panama City Cancer Institute
Cost: 8 patients died, 20 received an overdose of radiation
The series of incidents happened in the National Cancer Institute in Panama City in 2000. The software for radiation therapy planning produced by the American company Multidata Systems International made errors during calculating radiation doses.
Doctors could draw the position of protective metallic disks supposed to protect the body from radiation. The program allowed to manage four discs; doctors, however, expected to use five of them. They found a way to “outsmart” the system: to arrange the disks in a single block with a hole in the center. One thing they didn’t consider is that such an order wasn’t mentioned in the guidelines. The program calculated the amount of radiation depending on how the hole is drawn and in some cases produced a double dose of radiation.
Eight people died as a result of the mistake; 20 patients received the overdose of radiation. The doctors, who were to check the results before starting the radiation, were convicted of killing.
Bottom line
What cost of bugs in software testing are you ready to accept? Isn’t it better to detect some issues before you release the product with unexpected features that can cost you time, money, reputation, or even affect someone’s health? You already know the answer, and we know how to release bug-free software.
We value the privacy of each client and cannot share the information about the most serious bugs we have fixed. What we can do is to help you to release high-quality products. Leave a message and let’s discuss the cooperation.
