[Cryptography] Entropy is forever ...

[John Denker]

Clarification: The full, correct statement is:

  For any given distribution,                        [a]
  the entropy is a property of the distribution      [b]
  ... not of any particular string that may have 
  been drawn from such a distribution.               [c]

When people are in a hurry, common practice (but not good
practice) is to assert part [b] without being explicit about 
part [a].  An example is my message from 04/17/2015 11:59 AM.

Beware that in part [b], the word «the» must not be given
undue emphasis, because it is heavily modified by part [a],
and must not be taken out of context.  That is, we are
*not* talking about «the» one true distribution.  In
cryptography and many other situations, it is common to
have more than one distribution in use at the same time.

Even if you know the string, you don't know «the» distribution
from which it was drawn.  In a card game, the microstate
(i.e. the objective state of the cards) is the same for
everybody, but different players see the probabilities
differently, i.e. they are working with different
distributions.  For further discussion including a
diagram, see
  https://www.av8n.com/physics/thermo/entropy.html#sec-micro-macro

In crypto, it is virtually always the case that the sender
and the attacker are using wildly different distributions 
over plaintexts and keys.  The microstate is known to the
sender, whereas the attacker presumably has to guess.  The
microstate is the same for both, but the distributions are
different.

Therefore it is madness to speak of «the» entropy inherent
in a string.  It is not possible to ascertain or even define
«the» one true distribution, not using Turing machines, 
not using philosophy, not by any means whatsoever.

In the crypto business, more often than not, when people
talk about «the» distribution they are talking about the
distribution /as seen by the attacker/ ... but I am not
touting this as a reliable rule.  The smart approach is 
to identify explicitly the distribution you are talking
about ... or (!) to carry the distribution as an unbound
variable.

  Yes, I know one can find lots of references to the
  idea of «the» «inherent» entropy.  By the same token,
   -- Climate change is a hoax: 
    http://www.amazon.com/The-Greatest-Hoax-Conspiracy-Threatens/dp/1936488493
    http://www.conservapedia.com/Global_warming
   -- Dinosaurs coexisted with humans:
    http://creationmuseum.org/whats-here/exhibits/allosaur/
   -- The Apollo moon landings were fake:
    http://huzlers.com/buzz-aldrin-admits-apollo-11-moon-landings-fake-simply-set-see-tweet/
   -- Cigarettes are not addictive:
    http://www.nytimes.com/1994/04/15/us/tobacco-chiefs-say-cigarettes-aren-t-addictive.html
   -- Elvis is not dead:
    http://www.imdb.com/title/tt0119654/quotes
   -- et cetera..............

  Similarly, I am quite aware that wikipedia says that
  both energy and entropy are extensive variables:
    http://en.wikipedia.org/wiki/Intensive_and_extensive_properties#Examples
  However, it's just not true (not for energy or entropy),
  especially for smallish systems, where surface states 
  make a nontrivial contribution.  I hate to belabor the
  obvious, but wikipedia is not a reliable authority for
  the foundations of physics or information theory.  It
  doesn't try to be.  It explicitly doesn't want to be.