[Cryptography] VW/EPA tests as crypto protocols ?

[Henry Baker]

At 12:36 PM 9/24/2015, John Denker wrote:
>The general principle here is simple:
>
>*Measure the thing you care about.*
>
>To say the same thing the other way:
>
>Avoid measuring something that is only a proxy for the thing you care about.
>
>As soon as you start rewarding and/or regulating the proxy, it ceases to be a reliable measure.
>
>In this double-negative form it is known as Goodheart's law:
>
>https://en.wikipedia.org/wiki/Goodhart%27s_law
>
>To be sure, a proxy is often better than nothing, and may be a valuable hint, for instance when you are dealing with potentially catastrophic but hopefully rare events.
>
>Example: you want to regulate the strength of nuclear power plant components, airliner components, deep-sea oil well components, et cetera.
>
>However, you still *must* hold the operator accountable for bottom-line results including bottom-line safety.

I wasn't previously aware of "Goodhart's Law"; there is a more quantitative form of the same effect from control theory, but I don't know if it has a proper name.

Basically, if you're trying to control a number of variables having differing levels of uncertainty (e.g., variance), you will end up controlling the system so as to push all/most of the uncertainty into the variable you can measure the *least well*.

Thus, if you are trying to control some position (x,y,z), where measurement variances are ordered variance(x)<variance(y)<variance(z), then your feedback control system will unwittingly push all of the variance of the controlled position into the z coordinate.  Thus, if (x,y,z) is the position of some airplane or spacecraft, you will *crash* it precisely at the position (x,y), because you could measure x and y, but not z!  (The alternative requires more subtlety: minimize the variance of z *regardless* of x & y; you will land safely, but just not exactly where you wanted.)

Perhaps this effect should be called the "out of sight, out of mind" ("out of control" !?!) theorem.  Alternatively, you might call it "crashing the spacecraft near the lightpost, because that's where it's easiest to see" theorem.

Other obvious examples of Goodhart's Law are medicines that "treat the numbers" instead of "treating the disease", and educators who "teach to the test".  (And then there are institutions such as Harvard that don't even trust themselves to "teach to the test", and simply arrange to admit only those who can *already* pass the test with no further instruction required!)