Re: [PATCH 1/3] Make /dev/urandom scalable

From: Austin S Hemmelgarn
Date: Fri Sep 25 2015 - 15:08:09 EST

On 2015-09-25 07:41, Austin S Hemmelgarn wrote:
On 2015-09-24 16:14, Theodore Ts'o wrote:
On Thu, Sep 24, 2015 at 03:11:23PM -0400, Austin S Hemmelgarn wrote:
That is a startling result. Please say what architecture, kernel
version, dieharder version and commandline arguments you are using to
get 10% WEAK or FAILED assessments from dieharder on /dev/urandom.

I do not remember what exact dieharder version or command-line arguments
(this was almost a decade ago), except that I compiled it from source
myself, I do remember it was a 32-bit x86 processor (as that was
sadly all I
had to run Linux on at the time), and an early 2.6 series kernel
(which if I
remember correctly was already EOL by the time I was using it).

It might have been nice if you had said this from the beginning
instead of making an unqualified statement with the assumption that it
was applicable to kernels likely to be used today in non-obsolete
systems. Otherwise it risks generating a click-bait article on
Phoronix that would get people really worried for no good reason...
I sincerely apologize about this, I should have been more specific right
from the beginning (I need to get better about that when talking to
people, I'm so used to dealing with some of my friends who couldn't
event tell you the difference between RAM and a hard drive, think a bus
is only something you use for transportation, and get confused when I
try to properly explain even relatively simple CS and statistics concepts).

There was a bug a long, long time ago (which where we weren't doing
sufficient locking and if two processes raced reading from
/dev/urandom at the same time, it was possible that the two processes
would get the same value read out from /dev/urandom). This was fixed
a long time ago, though, and in fact the scalability problem which
Andi is trying to fix was caused by that extra locking that was
added. :-)

It's possible that is what you saw. I don't know, since there was no
reproduction information to back up your rather startling claim.
I don't think this was what I hit, I'm pretty sure I had serialized the
dieharder runs.

If you can reproduce consistent Dieharder failures, please do let us
know with detailed reproduction instructures.
Will do.
OK, just started a couple of runs in parallel using different generators using the following command line:
dieharder -a -m 32 -k 1 -Y 1 -g XXX
with one each for:
/dev/urandom (502)
AES_OFB (205)
glibc random() (039)
mt19937 (013)
The above command line will run all dieharder tests with 12800 psamples, using a higher than default precision, and re-running tests that return WEAK until it gets a PASS or FAIL. Even on the relatively fast (at least, fast for a desktop) system I'm running them on, I expect it will take quite some time to finish (although regardless of that I'm probably not going to be getting back to it until Monday).

Interestingly, based on what dieharder is already saying about performance, /dev/urandom is slower than AES_OFB (at least, on this particular system, happy to provide hardware specs if someone wants).

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature