read() syscall slowing down due to other threads?

[George Porter]

Hi all,

I've run into a weird phenomenon I was hoping someone could help me pin down.

I've got a multithreaded program which reads data off of 8 input
disks, and does some processing on it.  I have 8 Reader threads, each
of which reads data off of one of the eight input disks.  That data
gets passed to other threads, which do some processing (this is an 8
core machine).  If I do no or minimal processing in those other
threads, the read() calls go at the speed of the disks (~100 MBps).  I
measure that speed by taking a timestamp before the read syscall, then
afterwards, and dividing that into the read size (5MB).  All seems
good.

However, if I start doing more computation on those other threads, the
read() syscalls take longer to read the same amount of data,
eventually slowing down to 50 MBps (50% slower).  I've used
setaffinity() to isolate the Reader threads to one set of cores, and
the compute threads to a different set of cores, and so I don't think
it is CPU/scheduling interference.

Thoughts?  Has anyone run into this before?

Also, if you could CC me directly on any responses I would appreciate it.
Thanks, George
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/