[Cryptography] End-to-End Protocols and Wasp Nests

Tom Mitchell mitch at niftyegg.com
Mon Mar 10 01:58:20 EDT 2014

On Sun, Mar 9, 2014 at 12:10 PM, Bear <bear at sonic.net> wrote:

> On Sat, 2014-03-08 at 20:46 -0500, John Kelsey wrote:
> > Isn't this just combinatorial testing?

> It is exactly combinatorial testing.  I've never seen any
> really good frameworks to do it though.
A problem with combinatorial testing is the permutations get
too numerous very quickly.  Testing cycles move from hours
to weeks.

I would assert() that:
asserts() have value but if combinatorial testing is the
best test strategy change assert to liveassert() or some such
macro that does not collapse to nothing when running
code.   Same for bounds testing....

At some future time liveassert() can be edited to be a
normal assert() should performance analysis discover
the hit matters and the test is unneeded in production.

This moves some combinational testing into the
world of real user data and short circuits  N! type
combinational testing to a subset that matters.

Also when an urgent fix is pushed to plug a zero day
or some such error ship with asserts(true) and follow
up them disabled a bit later.

A risk is that a bogus assert exist in the code base

Testing is hard... one quote I think attributed to Don Knuth
is that TeX took 1/10th the time while individual functional
tests took 9/10.    i.e. His one year project was "finished" in

For those shipping product time to market is important so
ship early if you must but leave asserts live until such time
that they are discovered to be unnecessary.

  T o m    M i t c h e l l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.metzdowd.com/pipermail/cryptography/attachments/20140309/47bc801f/attachment.html>

More information about the cryptography mailing list