Tip:
Highlight text to annotate it
X
So now that you've seen how easy it is to build a random tester,
I hope that you build many of them in the future and most of the rest of this course is going to be spent
going into more details about how we build effective random testers.
But before we do that, what I'd like to do is address an issue that eventually confronts
almost everybody who works on random testing, which is that random testing gets no respect.
And what I mean by that is people often think that it's a really stupid way to test software.
For example, one of the classic references on software testing
is a book called The Art of Software Testing.
In this book, the author talks about random testing and here's what he has to say.
"Probably the poorest methodology of all is random random input testing...
What we look for is a set of thought processes that allow one to select
a set of test data more intelligently."
So basically what Myers is saying is random input testing is not a way that allows one
to select a set of test data intelligently.
And so my opinion and this is after writing at least a dozen testers
what he should've said is something like what we look for in testing is a set of thought processes
that allow one to randomly generate a set of test data more intelligently.
And so what we can do is get around this fact that
testing requires thought about the software under test.
But my experience and the experience of a lot of other people is that if we think hard
about a random test case generator that's as good or better in many cases
than thinking hard about generating the test cases themselves.
So another classic piece of work is a book chapter called Random Testing by Richard Hamlet.
And in this chapter he says, "Most criticism of random testing is really objection
to misapplication of the method using inappropriate input distribution."
And so what Hamlet is really talking about here is the input validity problem
that we've just been discussing and what he means is that if you ignore the input validity problem
and you just test randomly using completely random garbage, you're going to get a bad
impression of the method because it's not going to work very well.
You're going to experience a phenomenon that we talked about a little bit ago
where all of the test cases get rejected by very early parts of the software under test.
And so I would actually say what Hamlet said a little bit differently.
Most criticism in random testing is really objection to random testing done badly.
So let's take a look at one more quote.
So what the author of Testing for Zero Bugs has to say is, "The introduction of random testing
practically eliminated user bug reports on released back ends.
To our amazement, RIG (and this is their test case generator) was able to find over half of all bugs
reported by customers on the code generator in just one night of run time."
And so this clearly was written by somebody who did a really good job creating a random tester.
And also probably applied in the domain where random testing happened to work really well.
The rest of this Testing for Zero Bugs is worth reading as well,
and we will include a link to that in the supplemental material for the course.