Software Testing Myths

Developer Myth #1:  

"Given enough time, eventually we can get all the bugs out"

View graphs

Reality:  Testing and debugging remove only a portion of the defects regardless of time spent.
Testing has a "filter" effect.  Each test technique can be thought of as a "filter" that catches some of the defects.

Analogy:  Removing defects from software via testing is like removing fish from an aquarium with a net.
Imagine a net with holes one inch in diameter.  There are some small fish that you will never remove regardless how long you strain the water.

sketch of fishnet

Reality:  A bad product going in to test will yield a bad product coming out of test.

Developer Myth #2:

"The way to improve the quality of the software is to do lots of testing." or simply "testing and debugging improves quality."

Reality: "Testing to  improve quality is like standing on a scale to lose weight"
Testing is a measure of quality.  The number of defects you find indicates the quality of the product.

The following aphorism is well understood and frequently repeated in other engineering fields:
"You can't test quality into a product".

Analogy:  Building a house without building codes, blueprints, reviews, etc.
Imagine a bunch of construction guys and the architect meet in an empty lot with the customer and a big stack of lumber.   Then erecting a house in a spontaneous fashion.   When the customer looks dubiously at the finished house and complains that the roof doesn't look very secure the contractor replies "Don't worry, we'll just wait until it rains and then patch where it leaks."

It's probably obvious that an improvised patch to a leak does not improve the quality of the house. Defects are just a symptom of poor quality, a superficial manifestation.   Repairing the defect doesn't effect the underlying quality.  "Once a lemon always a lemon."

No other profession build products of uncontrolled quality then relies on testing (and defect repair) to improve the product quality.  (Partly because other fields have tangible products where rework has substantial materials costs.  Programmers have the luxury of an intangible product and then conclude that an undisciplined development process will be adequate).

Developer Myth #3 (derived myth):

Myth: "Producing high quality software takes longer."
Because of myths 1 and 2, developers conclude that producing high quality software takes longer than poor quality software.

Reality: A focus on high quality from the beginning reduces development time because there are fewer downstream defects.  (Cost escalation model shows that downstream defects are much more expensive).

View graphs

Why do these myths persist?  Most developers have never gathered the quantitative data about their own work to see the facts.

By contrast, companies that follow the "cleanroom" process (which emphasizes formal methods and reviews) are often able to completely skip the unit test phase. Their reviews are so effective there are no defects left to find in unit test.