Re: [PATCH] sunrpc: use better NUMA affinities

[Greg Banks]

Sent from my iPhone

On 30/07/2011, at 4:08, "J. Bruce Fields" <bfields@xxxxxxxxxxxx> wrote:

> On Fri, Jul 29, 2011 at 08:02:05PM +0200, Eric Dumazet wrote:
> > Le vendredi 29 juillet 2011 Ã 12:42 -0400, J. Bruce Fields a Ãcri 
> > t :
> > > On Thu, Jul 28, 2011 at 08:04:09PM +0200, Eric Dumazet wrote:
> > > > Use NUMA aware allocations to reduce latencies and increase  
> > > > throughput.
> > > >
> > > > sunrpc kthreads can use kthread_create_on_node() if pool_mode is
> > > > "percpu" or "pernode", and svc_prepare_thread()/svc_init_buffer()  
> > > > can
> > > > also take into account NUMA node affinity for memory allocations.
> > > ...
> > > > @@ -662,14 +675,16 @@ svc_set_num_threads(struct svc_serv *serv,  
> > > > struct svc_pool *pool, int nrservs)
> > > >        nrservs--;
> > > >        chosen_pool = choose_pool(serv, pool, &state);
> > > >
> > > > -        rqstp = svc_prepare_thread(serv, chosen_pool);
> > > > +        node = svc_pool_map_get_node(chosen_pool->sp_id);
> > > > +        rqstp = svc_prepare_thread(serv, chosen_pool, node);
> > >
> > > The only correct value for the third argument there is
> > > svc_pool_map_get_node(chosen_pool->sp_id), so let's have
> > > svc_prepare_thread() call that itself.
> >
> > I have no idea of what you mean ;)
> >
> > I need 'node' for the following kthread_create_on_node()
>
> Doh, of course--apologies.
>
> > > Seems OK otherwise.
> > >
> > > Any suggestions on how we should test this?
> >
> > I did tests on my machine, seems good.
> >
> > I checked that stacks were now correct using :
> > "echo t > /proc/sysrq-trigger"
>
> I was wondering more about good tests of nfsd's performance on numa;
> that might be more of a question for Greg.

To really show a big difference you need a much bigger box, or slower  
NUMA interconnects than today's. You also want network cards locally  
attached to each node and a metadata heavy (i.e. high rpc call rate)  
load.

Greg.--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/