[Cryptography] sha1sum speed

[Henry Baker]

At 02:54 PM 4/30/2016, Mark Steward wrote:
>Tell me you cached the files first.
>
>Mark
>
>On Sat, Apr 30, 2016 at 5:00 PM, Henry Baker <hbaker1 at pipeline.com> wrote:
>I just run Linux's 'sha1sum' on a number of very large files, and the calculation took significantly longer than I expected.
>
>'sha1sum' is only modestly faster on a very large file than copying the file.
>
>I noticed that
>1) the cpu meter wasn't pinned at 100%; and
>2) multiple cores weren't being fully utilized.
>
>BTW, I don't care about "SHA1", per se; I could just as easily have used "SHA256" or some other hash function.

After cacheing this large file (i.e., on the 2nd & subsequent timings),

sha1sum took 24 seconds.
sha3sum (default algorithm) took 54 seconds.
sha256sum took 54 seconds.
b2sum-i686-linux took 35.7 seconds.
b2sum-amd64-linux took 27.3 seconds.
cksum took 22.8 seconds.
cfv -C -tsha1 -f- took 19.5 seconds
cfv -C -tcrc -f- took 8.8 seconds
cfv -C -tmd5 -f- took 15.6 seconds

ALL of these timings are single-threaded on the same Ubuntu box.

Soooo, there are substantial differences in calculation times -- even among SHA1 implementations.

Off-hand, I'd say that 'cksum' and 'sha1sum' could use a little TLC to improve their performance.