Software Testing Myths
Developer Myth #1:
"Given enough time, eventually we can get all the bugs out"
View graphs
Reality: Testing and debugging remove only a portion of the
defects regardless of time spent.
Testing has a "filter" effect. Each test technique can be thought
of as a "filter" that catches some of the defects.
Analogy: Removing defects from software via testing is like
removing fish from an aquarium with a net.
Imagine a net with holes one inch in diameter. There are some
small fish that you will never remove
regardless how long you strain the water.
Reality: A bad product going in to test will yield a bad product
coming out of test.
Developer Myth #2:
"The way to improve the quality of the software is to do lots of
testing."
or simply "testing and debugging improves quality."
Reality: "Testing to improve quality is like standing on a scale
to lose weight"
Testing is a measure of quality. The number of defects you find
indicates the quality of the product.
The following aphorism is well understood and frequently repeated in
other engineering fields:
"You can't test quality into a product".
Analogy: Building a house without building codes, blueprints,
reviews, etc.
Imagine a bunch of construction guys and the architect meet in an empty
lot with the customer
and a big stack of lumber. Then erecting a house in a
spontaneous fashion. When the customer
looks dubiously at the finished house and complains that the roof
doesn't look very secure the
contractor replies "Don't worry, we'll just wait until it rains and
then patch where it leaks."
It's probably obvious that an improvised patch to a leak does not
improve the quality of the house.
Defects are just a symptom of poor quality, a superficial
manifestation. Repairing the defect
doesn't effect the underlying quality. "Once a lemon always a
lemon."
No other profession build products of uncontrolled quality then relies
on testing (and defect repair) to
improve the product quality. (Partly because other fields have
tangible products where rework has
substantial materials costs. Programmers have the luxury of an intangible
product and then conclude
that an undisciplined development process will be adequate).
Developer Myth #3 (derived myth):
Myth: "Producing high quality software takes longer."
Because of myths 1 and 2, developers conclude that producing high
quality software takes longer
than poor quality software.
Reality: A focus on high quality from the beginning reduces development
time
because there are fewer downstream defects. (Cost escalation
model shows that downstream
defects are much more expensive).
View graphs
Why do these myths persist? Most developers have never gathered
the quantitative data
about their own work to see the facts.
By contrast, companies that follow the "cleanroom" process
(which emphasizes formal methods and reviews)
are often able to completely skip the unit test phase.
Their reviews are so effective there are no defects
left to find in unit test.