Re: [RFC] NUMA futex hashing
From: Ravikiran G Thirumalai
Date: Tue Aug 08 2006 - 20:09:03 EST
On Tue, Aug 08, 2006 at 12:10:39PM +0200, Eric Dumazet wrote:
> On Tuesday 08 August 2006 11:57, Andi Kleen wrote:
> > Ravikiran G Thirumalai <kiran@xxxxxxxxxxxx> writes:
> > > Current futex hash scheme is not the best for NUMA. The futex hash
> > > table is an array of struct futex_hash_bucket, which is just a spinlock
> > > and a list_head -- this means multiple spinlocks on the same cacheline
> > > and on NUMA machines, on the same internode cacheline. If futexes of two
> > > unrelated threads running on two different nodes happen to hash onto
> > > adjacent hash buckets, or buckets on the same internode cacheline, then
> > > we have the internode cacheline bouncing between nodes.
> > When I did some testing with a (arguably far too lock intensive) benchmark
> > on a bigger box I got most bouncing cycles not in the futex locks itself,
> > but in the down_read on the mm semaphore.
> This is true, even with a normal application (not a biased benchmark) and
> using oprofile. mmap_sem is the killer.
Not if two threads of two different process (so no same mmap_sem) hash onto
futexes on the same cacheline. But agreed, mmap_sem needs to be fixed too.
If everyone agrees on a per-process hash table for private futexes, then
we will work on that approach.
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/