Re: [PATCH v7 1/4] spinlock: A new lockref structure for locklessupdate of refcount

[Waiman Long]

On 08/29/2013 11:54 PM, Linus Torvalds wrote:
> On Thu, Aug 29, 2013 at 8:12 PM, Waiman Long<waiman.long@xxxxxx>  wrote:
> > On 08/29/2013 07:42 PM, Linus Torvalds wrote:
> > > Waiman? Mind looking at this and testing? Linus
> > Sure, I will try out the patch tomorrow morning and see how it works out for
> > my test case.
> Ok, thanks, please use this slightly updated pCMPXCHG_LOOPatch attached here.

I tested your patch on a 2-socket (12 cores, 24 threads) DL380 with 
2.9GHz Westmere-EX CPUs, the test results of your test program (max 
threads increased to 24 to match the thread count) were:

with patch = 68M
w/o patch = 12M

So it was an almost 6X improvement. I think that is really good. A 
dual-socket machine, these days, shouldn't be considered as a "BIG" 
machine. They are pretty common in different organizations.

I have reviewed the patch, and it looks good to me with the exception 
that I added a cpu_relax() call at the end of while loop in the 
CMPXCHG_LOOP macro.

I also got the perf data of the test runs with and without the patch.

With patch:

 29.24%    a.out  [kernel.kallsyms]    [k] lockref_get_or_lock
 19.65%    a.out  [kernel.kallsyms]    [k] lockref_put_or_lock
 14.11%    a.out  [kernel.kallsyms]    [k] dput
  5.37%    a.out  [kernel.kallsyms]    [k] __d_lookup_rcu
  5.29%    a.out  [kernel.kallsyms]    [k] lg_local_lock
  4.59%    a.out  [kernel.kallsyms]    [k] d_rcu_to_refcount
    :
  0.13%    a.out  [kernel.kallsyms]    [k] complete_walk
    :
  0.01%    a.out  [kernel.kallsyms]    [k] _raw_spin_lock

Without patch:

 93.50%    a.out  [kernel.kallsyms]    [k] _raw_spin_lock
  0.96%    a.out  [kernel.kallsyms]    [k] dput
  0.80%    a.out  [kernel.kallsyms]    [k] kmem_cache_free
  0.75%    a.out  [kernel.kallsyms]    [k] lg_local_lock
  0.48%    a.out  [kernel.kallsyms]    [k] complete_walk
  0.45%    a.out  [kernel.kallsyms]    [k] __d_lookup_rcu

For the other test cases that I am interested in, like the AIM7 
benchmark, your patch may not be as good as my original one. I got 1-3M 
JPM (varied quite a lot in different runs) in the short workloads on a 
80-core system. My original one got 6M JPM. However, the test was done 
on 3.10 based kernel. So I need to do more test to see if that has an 
effect on the JPM results.

Anyway, I think this patch is good performance-wise. I remembered that 
awhile ago that an internal reported a lock contention problem in dentry 
involving probably complete_walk(). This patch will certainly help for 
that case.

I will do more investigation to see how to make this patch work better 
for my test cases.

Thank for taking the effort in optimizing the complete_walk() and 
unlazy_walk() function that are not in my original patch. That will make 
the patch work even better under more circumstances. I really appreciate 
that.

Best regards,
Longman



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/