[Cryptography] sha1sum speed

[Tom Mitchell]

On Sat, Apr 30, 2016 at 9:00 AM, Henry Baker <hbaker1 at pipeline.com> wrote:

> I just run Linux's 'sha1sum' on a number of very large files, and the
> calculation took significantly longer than I expected.
>
> 'sha1sum' is only modestly faster on a very large file than copying the
> file.
>
 ....

> 1) the cpu meter wasn't pinned at 100%; and
> 2) multiple cores weren't being fully utilized.


This sounds like disk I/O congestion.
To speed it up look to mmap() of the input file
and bypass the file system block cache.

Tinker with files of increasing size and see where the edges
are.  Simple benchmark sets like lmbench, dd and bonnie
are worth a look.

trace or ptrace can let you see how the file is opened as well
as a source code inspection.

Another trick is to run a program that uses calloc() calls to claim
and use memory allowing the system to read the input files
into free pages of memory.  Not too much memory but a lot.

Once disk I/O dominates the choice of hash or sum is no longer
a worry.   Fast SSD devices move the bar a lot and will be the
most common future first layer storage.



-- 
  T o m    M i t c h e l l
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.metzdowd.com/pipermail/cryptography/attachments/20160430/9d21e5dd/attachment.html>