[Cryptography] Rubber-hose resistance?

John Gilmore gnu at toad.com
Tue Dec 26 03:41:14 EST 2017


One follow-up re ATA Secure Erase: Some SSD's really do not implement
it correctly.  One even claimed to have executed it, but really did
absolutely nothing!  Others left parts of the prior data in the flash
chips, accessible to forensic probing.  I pointed out the 2010/2011
USENIX paper about this to the list on 19 June 2014 (see below).  Does
anyone know of any updated papers that test Secure Erase on more
modern SSDs, hard drives, and SDcards?  Or any improvements on USB
memory sticks to allow them to be securely erased?

	John

Message-Id: <201406200010.s5K0A8eo013740 at new.toad.com>
To: Thierry Moreau <thierry.moreau at connotech.com>
In-reply-to: <53A25FC7.5040105 at connotech.com> 
References: <53A25FC7.5040105 at connotech.com>
Comments: In-reply-to Thierry Moreau <thierry.moreau at connotech.com>
 message dated "Thu, 19 Jun 2014 03:57:59 -0000."
Date: Thu, 19 Jun 2014 17:10:08 -0700
From: John Gilmore <gnu at toad.com>
X-Mailman-Approved-At: Thu, 19 Jun 2014 20:15:51 -0400
Cc: cryptography at metzdowd.com
Subject: Re: [Cryptography] Shredding a file on a flash-based file system?
X-BeenThere: cryptography at metzdowd.com
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: The Cryptography and Cryptography Policy Mailing List
 <cryptography.metzdowd.com>
List-Unsubscribe: <http://www.metzdowd.com/mailman/options/cryptography>,
 <mailto:cryptography-request at metzdowd.com?subject=unsubscribe>
List-Archive: <http://www.metzdowd.com/pipermail/cryptography/>
List-Post: <mailto:cryptography at metzdowd.com>
List-Help: <mailto:cryptography-request at metzdowd.com?subject=help>
List-Subscribe: <http://www.metzdowd.com/mailman/listinfo/cryptography>,
 <mailto:cryptography-request at metzdowd.com?subject=subscribe>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Errors-To: cryptography-bounces+gnu=toad.com at metzdowd.com
Sender: "cryptography" <cryptography-bounces+gnu=toad.com at metzdowd.com>

> Let's advance the field! Thank in advance.

We may try to advance, but we're standing on ground that's moving
backwards at a pretty good clip.

> Maybe the best answer requires experimentation with a specific 
> combination of solid state drive, controller, driver, and file system 
> (ext2 most likely). That is, behave like an enemy (law enforcement 
> friend) chasing the secret data after the zeroization attempt, except 
> that I known the exact data values to look for. A low-level read-only 
> access to the block device is the basic facility for this 
> unsophisticated experiment. Anybody attempted this?

Oh yes.

With "SSD" drives these researchers filled them with patterned data,
then tried erasing them various ways (including the "security erase whole
drive" ATA command).  They then took the drives apart and read out the
contents of each flash chip.  The results were really ugly.  I believe
this is the definitive paper so far:

  https://www.usenix.org/legacy/events/fast11/tech/full_papers/Wei.pdf
  http://cseweb.ucsd.edu/users/m3wei/assets/pdf/FMS-2010-Secure-Erase.pdf

With "USB memory sticks" the situation is even worse.  They have no
"secure erase whole drive" command specified, nor does the USB Mass
Storage command set provide any way to ask that individual blocks be
erased.

With both types, they are traditionally "overprovisioned" with more
blocks than are advertised, both so that they can continue to operate
after some blocks fail, and so that they can still allow writes into
completely full drives.  (Flash can only erase in "erase blocks" which
are typically dozens of blocks.  And they can't overwrite existing
blocks, they can only write to already-erased blocks.)  When erasing
an erase block, all the still-needed data from that erase block needs
to be copied to blocks in some other erase block, so you end up with
multiple copies of each block, scattered around in arbitrary places on
the flash chips.  And after you "overwrite" a block from the
interface, the data that used to be in that block is still there in
the flash chip; your new data is in a different block somewhere.

Each flash-based device has a "controller" chip with a CPU and
proprietary firmware that manages this process, maintaining free-lists
and damaged-lists and the mappings between block numbers on the
interface to which blocks of flash actually contain that data.  This
firmware is often buggy, comes with unknown provenance, and typically
has terrible performance under some conditions (that vary from drive
to drive).  Unfortunately most were tested only with FAT filesystems,
so more complicated or performant filesystems may interact badly with
their firmware.

Furthermore, you have no idea what flash chips nor controller are in
your package, even if it comes from a reputable manufacturer.  Bunnie
of Chumby fame had manufacturing problems with SD cards, so chased 
down his sources of supply.  Turned out many were counterfeit, and even
the "name brand" ones with identical labeling often had controllers
and flash chips from different vendors inside:

  http://www.bunniestudios.com/blog/?page_id=1022

The OLPC project, which was the first to ship laptops that had flash
and no disk drives, tested many flash chips and SD cards to
destruction.  They found a huge variety of provenance and performance
as well:

  http://lists.laptop.org/pipermail/devel/2010-August/029684.html

  "A batch of 2GB class 2 microSD cards obtained a year ago from a
  particular manufacturer averaged around 10 TB written before
  failing, with few transient errors. A batch of 2GB class 2 microSD
  cards from the same manufacturer today failed with more than half
  corrupting their FS after only 1TB of writes. The devices wear out
  around 2-4 TB of writes."

>						I would suspect that 
> the most fundamental optimization (not allocating new space when a write 
> request only turns "1" bits to "0") is applied effectively despite this 
> involved logic arrangement.

This basically never happens.  First, you don't know whether the
actual flash chip is storing your data "true" or "inverted", so you
don't know if 1->0 or 0->1 is the fundamental write operation.  [If it
was smart, it would decide to write each block either true or
inverted, with a flag saying which, by counting the number of erased bits
flipped in true versus inverted writes, thereby making the chip last
longer.  But they probably aren't smart.]  Second, each block written
is written with error checking and control bits, which are a hash
value of the contents of the block.  In these bits you are almost
guaranteed to turn a 0 to a 1 even if the contents of the block only
turn 1's to 0's.  So EVERY write to a block of managed flash involves
allocating a new block and putting the old block on the "to be erased"
list.  The only place that flash controllers can probably do the trick
of writing a block multiple times by only turning 1's to 0's, is in
its own internal data structures that keep track of which blocks are
in use.

It would be an interesting project to reverse-engineer a flash
controller's microcode.  Most of the ones in SD cards seem to be ARM
chips, which the free world has good tools for investigating, once you
figure out how to read out the microcode.  Or even more interesting to
write some GPL'd flash controller microcode, whose performance you
could understand and/or improve.

Perry said:
> This is also true of modern hard disks -- 

Modern hard drives do remap bad sectors automatically to other
sectors, but they still have a *very strong* tendency to overwrite an
old sector's data when you overwrite it from the interface.  And,
modern densities leave little or no remnant of the old data, unlike
the generations of drives that Peter Gutmann studied.  However, your
OS's filesystem may not overwrite *files* in place, so the "shred"
command may or may not be useful depending what filesystem you are
using.

Also, modern drives contain the ATA "Security Erase" command that will
erase the entire user-accessible parts of the drive in one pass, with
some doing "Security Erase Enhanced" that also zeroes all the remapped
sectors and user-inaccessible areas of the drive.  This command is not
only faster than shipping terabytes of zeroes across the bus, but it's
likely to be more resistant against subsequent microscopic analysis.
(But: after erasing a drive, I still read the whole thing back and
compare for all zeroes.  This guards slightly against buggy erase
firmware, but not against smart malevolent firmware.)

Whenever I cross the US border with a disk drive, I run the Security
Erase command on it first.  Go ahead, bastards, search my laptop, make
my day!  Then on the other side, I reinstall Linux from a virgin
distribtion image on SD or USB.  "hdparm --security-help" will give
you the commands.  First lock the drive with a password like "foo",
then erase it with the same password:

  hdparm --security-set-pass foo /dev/sdz && hdparm --security-erase-enhanced foo /dev/sdz &

Don't forget that final & or your terminal will hang for the duration
of the erase (half an hour or more).  ^Z does not interrupt it.

If you boot the laptop from the "live" distribution image, you can open a
terminal and issue this command against your main hard drive (/dev/sda).
In theory, once the erase command starts, the drive will not respond to
any other commands until you complete the erase, even if power is lost
partway through.  This is probably not as well tested as the basic erasure.

There exists a new category of "hybrid" hard drives that contain flash
chips, and transparently store a tiny fraction of the hard drive's
contents in the flash chip to improve the average access time.  Here's
an example:

  http://www.newegg.com/Product/Product.aspx?Item=N82E16822178380

(The customer reviews are interesting: apparently all the bugs aren't
out of the more complex firmware yet.  And the firmware seems oriented
toward flash-cacheing the files that are read right after the drive is
reset, e.g. at boot time, so your machine will visibly boot faster and
you will notice the speedup and smile.)  

For data security (ability to overwrite or erase), these drives are
not your friend.  Luckily these drives come at a premium price (as
well as being marketed as containing flash), so you can avoid them
if you care.

	John

  
_______________________________________________
The cryptography mailing list
cryptography at metzdowd.com
http://www.metzdowd.com/mailman/listinfo/cryptography



More information about the cryptography mailing list