[Cryptography] "Is FIPS 140-2 Actively harmful to software?"

[ianG]

On 24/06/2014 03:40 am, Theodore Ts'o wrote:
> On Mon, Jun 23, 2014 at 07:44:44PM -0400, John Kelsey wrote:
>>
>> That audit is only meaningful until the developers start changing
>> the code.  A code audit of the current version of the library
>> doesn't give anyone much assurance (or shouldn't) about later
>> versions of the library.  If you want the assurance of the audit,
>> you can't change the code!  I don't really see any way around that.
>> (Go ask current users of Skype about that.)
> 
> So if the audit is **perfect** then it doesn't matter.


There is no practical or reasonable audit in our space where this is
even remotely true.  I grant it is possible, and the aerospace folks
sometimes claim it, but it isn't a reasonable thing to assume, as you do
below.


> The library
> with the "goto fail" bug also passed the audit (just as the Taiwan
> Personal Certificate hardware token had passed the FIPS certification
> audit), and then the problem is the "don't even touch a line of this
> code, not even a comment", pretty much guarantees that no developer
> will look at the code after it has been audited, lest they result in
> the company getting charged hundreds of thousands of dollars.
> 
> Of course, this doesn't stop the bad guys from looking at the code,
> and finding entertaining problems like the "goto fail" bug.
> 
> So the question is does the audit actually do enough good that it's
> worth freezing all further development activity on the library?  It's
> true that further development could introduce bugs ---- but code clean
> ups can actually find and fix problems, too.
> 
> There are no easy answers here, agreed.  But one would think that if
> you've paid hundreds of thousands of dollars, and the code gets a
> "pass", you should have some assurance that the code doesn't have
> horrendous bugs in it.  If not, is it worth paying that money and
> freezing any cleanup activity?


Right.  That is the question.  Having worked both sides of this
equation, I'd say (and said [0]) that the answer is No.

As you correctly point out, the damage done by freezing is alone enough
to cause any benefit to disappear.  However there are other things that
need to be taken into account.

Firstly, the only people who know what audit means are the people doing
it, and that doesn't necessarily mean all of them, all of the time.  The
consequence of this is that it is almost certain that the outside
doesn't understand what it means, and it is almost certain that the true
requirements of the beneficiary are not met by the audit.

(To see this, consider a Spencarian argument:  year 0 when everything
was good, but nobody understood.  Then in year 1 there is a delta of
change, and so on for many years.  After a few several cycles,
self-fulfilling feedback loops take over.  Good disappears for the
customer, but costs do not.)

However, there is some value in doing an audit, once.  And then carrying
on without.  In that, if you can pass, it has done something to improve
your capability to achieve something.  Once you've mined that, you
should drop it.

That isn't to say, you should do an audit.  It is to say that given your
particular circumstances, you could kick your team into a higher plane
of existence.  Or maybe not.  It all depends.



> (Other than so you can sell into the
> US Government market, that is....)


Right.  The requirer of the audit is the one that should benefit.  Now,
if they are working to the book, then they get what they asked for,
which is something.

But, is it likely that they are working to the book?  In a fast moving
software world, are the various USG users of (say) OpenSSL still using
the FIPS approved versions?

I don't think so.  So, as another consequence of the uncertainties, even
the USG is mostly upgrading its product as and when releases come along.
 In our space.


>> The ideal situation w.r.t. a software validation would include a
>> digital signature on the source code, right?  And then any change to
>> the source code, or the part covered by the signature/audit, would
>> automatically invalidate it.  You could imagine some ways to make
>> extending the validation to include some changes more economical,
> 
> Well, the ideal situation is that the software validation would cover
> a specific git commit.  That way futher validations could look at the
> code changes since that particular git commit, instead of starting
> from scratch.


Which leads to .. the dev team.  If you are that close to the changes
coming in, and you're responding to each of them, you may as well *be
the dev team*.

In which case, you the dev team is doing self-audit, and the external
audit isn't necessary.  In which case, the only thing you need is for
your own procedures to be fully disclosed, so that an external validator
can come to a reliance decision.


> And ideally, if the FIPS labs are going to charge
> hundreds of thousands of dollars, they should be willing to pay a
> bounty of say, $50,000 per security bug that they didn't catch, with a
> trusted third party validating whether a security bug report was valid
> or not.  And of course, they shouldn't charge to validate the fix.


Hope and dreams.  That is not how audit works.  Audit provides no
guarantee that there aren't any horrendous bugs in it even on day 0 with
the approved version.

(To repeat, only those doing the audit know what it means, in general.
Therefore, in general, your idea of what it means is wrong.  In general.)


> And maybe the certification should be paid for by the insurance
> company, with the companies paying insurance to cover the economic
> damages for any missed security vulnerability.
> 
> But as it stands, the FIPS labs are basically a tax on companies that
> want to sell to the US government, and presumably that means the
> prices for selling into the US government market are jacked up
> accordingly.  Which means the economic incentives are all broken, and
> the people who end up getting fleeced are the US Taxpayers....


Yep.  That's one of the self-fulfilling feedback loops.



iang



[0] http://financialcryptography.com/mt/archives/001126.html