Tip:
Highlight text to annotate it
X
What we want is to build this system-level random testers
as early as possible in the development process, and there are a number of reason for this.
First of all what we'd like to do is start off with a simple version of our system
that doesn't implement very much functionality and then we want to use sort of a weak fuzzer.
That is to say, we're going to test it with values that are maybe perhaps not that interesting.
And the good thing about this combination is
these weak random tests probably are going to find some flaws.
What they are not going to do is flood our developers with huge numbers of bugs
like might happen when we use an extremely strong random tester.
There is sort of no easier way to demoralized software developers
than to hand them a really big pile of bugs--nobody wants that.
What they're gong to start to do is ignore those bugs and get back to getting work done.
What you want to do is give people a slow but steady stream of important bug reports
and let them fix this as they go on, and that's another reason why it's nice
to give people with continuous stream of bug reports,
which is that what these do is help show us flaws in our software.
They help show us interfaces that we don't understand.
They help show us modules that end up being extremely weak for one reason or another,
and it basically helps us better understand where our software development effort is going wrong.
Now, if instead of giving people maybe a couple of bug reports a week for a year,
we give them a hundred bug reports in the very end
All they're going to do is triage to find the five most critical bugs, and fix them using hacks.
So nobody learns anything. Everybody is angry. Nobody is happy.
Rather what we like to have done is have been doing random testing all along
and using it to spot weakness in our software.
The other thing that happens is, as our software evolves
to be more robust as we move toward releasing it,
we're evolving our random tester to be stronger and stronger.
That is to say, maybe this week we have a feature where
we generate a new kind of random input that we haven't generated before.
Also it's going to generate some bug report, and we'll fix them,
and our software evolves to be more robust.
If we keep doing them not just over weeks but over years, what we'll end up with is
a random tester and a system that have sort of gone through this co-evolution process
where they both become much stronger.
That is to say, we've evolved in extremely sophisticated random tester,
and we've also evolved the system that gives robust with respect
to the kind of vaults it can be triggered by that random tester.
What I firmly believe is, and of course I can't prove this and you're free to disagree,
is that if, for example, Microsoft had done this from the beginning, Adobe had done this from the beginning,
and these other companies that end up with lots of security vulnerabilities
had fuzzed their products all the way through the development chain
they'd end up with far fewer of this sort of nasty crashes and critical security vulnerabilities
that they are always scrambling to patch, and that anybody with a fuzzer
seems to be able to find without a whole lot of effort.
Or at least that's been the case in the past, and it's possible now that with more widespread user
of more aggressive fuzzers with that kind of error of easy security bugs in popular products
maybe is hopefully starting to kind of tail off.