Re: How can I optimize a process on a NUMA architecture(x86-64 specifically)?

[Andrew Theurer]

On Sunday 23 May 2004 09:28, Andi Kleen wrote:
> On Sat, May 22, 2004 at 05:28:09PM -0700, Bryan O'Sullivan wrote:
> > On Fri, 2004-05-21 at 16:42, Brett E. wrote:
> > > Right now, 5 processes are running taking up a good deal of the CPU
> > > doing memory-intensive work(cacheing) and I notice that none of the
> > > processes seem to have CPU affinity.
> >
> > I don't know what kind of system you're running on, but if it's a
> > multi-CPU Opteron, it is normally a sufficient fudge to just use
> > sched_setaffinity to bind individual processes to specific CPUs.  The
> > mainline kernel memory allocator does the right thing in that case, and
> > allocates memory locally when it can.
> >
> > You can use the taskset command to get at this from the command line, so
> > you may not even need to modify your code.
>
> Linus also merged the NUMA API support into mainline now with 2.6.7rc1, so
> you can use numactl for more finegrained tuning.

FYI Brett, some Opteron systems have a BIOS option to interleave memory.  If 
you are going to make use of NUMA, I think you want to not interleave.

Also, if you have a 25% imbalance within a domain/node, the scheduler can have 
a tendency to bounce around a task for fairness.  That might be why you are 
seeing little/no affinity to a cpu (even top might be causing some of this).   
Not sure what the threshold is between domains/nodes, but I am curious if it 
still happens with CONFIG_NUMA on.  If these are long lived cpu bound 
processes, I would try to have the number of processes be a multiple of the 
number of cpus.

-Andrew Theurer
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/