Re: [RFC PATCH 1/2] percpu_counter: Allow falling back to global counter on large system

[Waiman Long]

On 03/07/2016 01:24 PM, Christoph Lameter wrote:
> On Fri, 4 Mar 2016, Waiman Long wrote:
>
> > This patch provides a mechanism to selectively degenerate per-cpu
> > counters to global counters at per-cpu counter initialization time. The
> > following new API is added:
> >
> >    percpu_counter_set_limit(struct percpu_counter *fbc,
> >                             u32 percpu_limit)
> >
> > The function should be called after percpu_counter_set(). It will
> > compare the total limit (nr_cpu * percpu_limit) against the current
> > counter value.  If the limit is not smaller, it will disable per-cpu
> > counter and use only the global counter instead. At run time, when
> > the counter value grows past the total limit, per-cpu counter will
> > be enabled again.
> Hmmm... That is requiring manual setting of a limit. Would it not be
> possible to completely automatize the switch over? F.e. one could
> keep a cpumask of processors that use the per cpu counters.
>
> Then in the fastpath if the current cpu is a member increment the per cpu
> counter. If not do the spinlock thing. If there is contention add the
> cpu to the cpumask and use the  per cpu counters. Thus automatically
> scaling for the processors on which frequent increments are operating.
>
> Then regularly (once per minute or so) degenerate the counter by folding
> the per cpu diffs into the global count and zapping the cpumask.
>
> If the cpumask is empty you can use the global count. Otherwise you just
> need to add up the counters of the cpus set in the cpumask.

I have modified the patch to try that out. However, that doesn't yield 
that much of improvement in term of performance and it slows down the 
percpu fast path a bit. So I am going to focus on my existing patch 
first and think about that later.

Cheers,
Longman