[RFC Patch 0/3] mm/slub: reduce contention for per-node list_lock for large systems

[Feng Tang]

Hi All,

Please help to review the ideas and patches, thanks!

Problem
-------

0Day bot found performance regression of 'hackbench', related with
slub's per-node 'list_lock' contention [1]. The same lock contention
is also found when running will-it-scale/mmap1 benchmark on rather
big system of 2 sockets with 224 CPUs, where the lock contention can
take up to 76% of cpu cycles.

As the trend is one processor (socket) will have more and more cpu
cores, the contention can be more severe, and we need to tackle it
sooner or later. 

Possible mitigations
--------------------

There are 3 directions we can try, they don't have dependency over
each other and can be taken separately or put together:

1) increase the order of each slab (including changing the max slub
   order from 3 to 4)
2) increase number of per-cpu partial slabs
3) increase the MIN_PARTIAL and MAX_PARTIAL to let each node have
   more (64) partial slabs in maxim 

Regarding reducing the lock contention and improving peformance,
#1 is the most efficient way, #2 second it.

Please be noted that the 3 patches are just for showing the idea
separately to get review and comments first, and NOT targeting for
merge. Patch 2 even can't apply upon patch 1.

A similar regression related to 'list_lock' contention was found when
testing 'hackbench' with new 'eevdf' scheduer patchset, and a rough
combination of these patches cure the performance drop [2].


Performance data
----------------

We have showed some rough performance data in previous discussion:
https://lore.kernel.org/all/ZO2smdi83wWwZBsm@feng-clx/

Following is performance data for using 'mmap1' case of 'will-it-scale'
and 'hackbench' mentioned in [1]. For 'mmap1' case, we run  3
configurations with parallel test threads of 25%, 50% and 100% of
number of CPUs

The test HW is a 2 socket Sapphire Rapids server (112 cores / 224
threads) + 256 GB DRAM, the base kernel is vanilla kernel v6.5.

1) order increasing patch

   * will-it-scale/mmap1:
   
   		     base                      base+patch
   wis-mmap1-25%    223670           +33.3%     298205        per_process_ops
   wis-mmap1-50%    186020           +51.8%     282383        per_process_ops
   wis-mmap1-100%    89200           +65.0%     147139        per_process_ops
   
   Take the perf-profile comparasion of 50% test case, the lock contention
   is greatly reduced:
   
         43.80           -30.8       13.04       pp.self.native_queued_spin_lock_slowpath
         0.85            -0.2        0.65        pp.self.___slab_alloc
         0.41            -0.1        0.27        pp.self.__unfreeze_partials
         0.20 ±  2%      -0.1        0.12 ±  4%  pp.self.get_any_partial
   
   * hackbench: 
   
   		     base                      base+patch
   hackbench	    759951           +10.5%     839601        hackbench.throughput
   
   perf-profile diff:
        22.20 ±  3%     -15.2        7.05        pp.self.native_queued_spin_lock_slowpath
         0.82            -0.2        0.59        pp.self.___slab_alloc
         0.33            -0.2        0.13        pp.self.__unfreeze_partials
   
2) increasing per-cpu partial patch

   The patch itself only makes the per-cpu partial number 2X, and for
   better analysis, the 4X case is also profiled

   * will-it-scale/mmap1:

		  base             base + 2X patch        base + 4X patch
   wis-mmap1-25	 223670    +12.7%     251999     +34.9%     301749    per_process_ops
   wis-mmap1-50	 186020    +28.0%     238067     +55.6%     289521    per_process_ops
   wis-mmap1-100  89200    +40.7%     125478     +62.4%     144858    per_process_ops
   
   Take the perf-profile comparasion of 50% test case, the lock contention
   is greatly reduced:
   
        43.80           -11.5       32.27           -27.9       15.91   pp.self.native_queued_spin_lock_slowpath
   
   * hackbench (no obvious improvment)
                 
		 base             base + 2X patch        base + 4X patch
   hackbench	759951      +0.2%    761506      +0.5%     763972     hackbench.throughput
 
 3) increasing per-node partial patch 

    The patch effectively change the MIN_PARTIAL/MAX_PARTIAL to from
    5/10 to 64/128. 

    * will-it-scale/mmap1:

   		     base                      base+patch
    wis-mmap1-25%    223670            +0.2%     224035        per_process_ops
    wis-mmap1-50%    186020           +13.0%     210248        per_process_ops
    wis-mmap1-100%    89200           +11.3%      99308        per_process_ops

  4) combination patches


			base	            base+patch-3       base+patch-3,1        base+patch-3,1,2
     wis-mmap1-25%	223670      -0.0%     223641     +24.2%     277734    +37.7%     307991     per_process_ops
     wis-mmap1-50%	186172     +12.9%     210108     +42.4%     265028    +59.8%     297495     per_process_ops
     wis-mmap1-100%	 89289     +11.3%      99363     +47.4%     131571    +78.1%     158991     per_process_ops


Make the patch only affect large systems
----------------------------------------
      
In real world, there are different kinds of platforms which has
different useage cases, large systems with huge numbers of CPUs
usually comes with huge memory, and there are also small devices
with limited memory, whch may care more about memory footprint.

So the idea is to treat them separately, keep the current
order/partial settings for system with small number of CPUs, and
scale those settings according to CPU numbers. (there is similar
handling in slub code already). Though aggressive idea is to bump
them all together.

[1]. https://lore.kernel.org/all/202307172140.3b34825a-oliver.sang@xxxxxxxxx/
[2]. ttps://lore.kernel.org/lkml/ZORaUsd+So+tnyMV@chenyu5-mobl2/

Thanks,
Feng

Feng Tang (3):
  mm/slub: increase the maximum slab order to 4 for big systems
  mm/slub: setup maxim per-node partial according to cpu numbers
  mm/slub: double per-cpu partial number for large systems

 mm/slub.c | 7 +++++++
 1 file changed, 7 insertions(+)

-- 
2.27.0