Re: [PATCH] slub: fix slub segmentation

[Chengming Zhou]

On 2024/4/2 11:10, Ming Yang wrote:
> When one of numa nodes runs out of memory and lots of processes still
> booting, slabinfo shows much slub segmentation exits. The following
> shows some of them:
> 
> tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs>
> <num_slabs> <sharedavail>
> kmalloc-512        84309 380800   1024   32    8 :
> tunables    0    0    0 : slabdata  11900  11900      0
> kmalloc-256        65869 365408    512   32    4 :
> tunables    0    0    0 : slabdata  11419  11419      0
> 
> 365408 "kmalloc-256" objects are alloced but only 65869 of them are
> used; While 380800 "kmalloc-512" objects are alloced but only 84309
> of them are used.
> 
> This problem exits in the following senario:
> 1. Multiple numa nodes, e.g. four nodes.
> 2. Lack of memory in any one node.
> 3. Functions which alloc many slub memory in certain numa nodes,
> like alloc_fair_sched_group.
> 
> The slub segmentation generated because of the following reason:
> In function "___slab_alloc" a new slab is attempted to be gotten via
> function "get_partial". If the argument 'node' is assigned but there
> are neither partial memory nor buddy memory in that assigned node, no
> slab could be gotten. And then the program attempt to alloc new slub
> from buddy system, as mentationed before: no buddy memory in that
> assigned node left, a new slub might be alloced from the buddy system
> of other node directly, no matter whether there is free partil memory
> left on other node. As a result slub segmentation generated.
> 
> The key point of above allocation flow is: the slab should be alloced
> from the partial of other node first, instead of the buddy system of
> other node directly.
> 
> In this commit a new slub allocation flow is proposed:
> 1. Attempt to get a slab via function get_partial (first step in
> new_objects lable).
> 2. If no slab is gotten and 'node' is assigned, try to alloc a new
> slab just from the assigned node instead of all node.
> 3. If no slab could be alloced from the assigned node, try to alloc
> slub from partial of other node.
> 4. If the alloctation in step 3 fails, alloc a new slub from buddy
> system of all node.

FYI, there is another patch to the very same problem:

https://lore.kernel.org/all/20240330082335.29710-1-chenjun102@xxxxxxxxxx/

> 
> Signed-off-by: Ming Yang <yangming73@xxxxxxxxxx>
> Signed-off-by: Liang Zhang <zhangliang5@xxxxxxxxxx>
> Signed-off-by: Zhigang Wang <wangzhigang17@xxxxxxxxxx>
> Reviewed-by: Shixin Liu <liushixin2@xxxxxxxxxx>
> ---
> This patch can be tested and verified by following steps:
> 1. First, try to run out memory on node0. echo 1000(depending on your memory) > 
> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages.
> 2. Second, boot 10000(depending on your memory) processes which use setsid 
> systemcall, as the setsid systemcall may likely call function 
> alloc_fair_sched_group.
> 3. Last, check slabinfo, cat /proc/slabinfo.
> 
> Hardware info:
> Memory : 8GiB
> CPU (total #): 120
> numa node: 4
> 
> Test clang code example:
> int main() {
>     void *p = malloc(1024);
>     setsid();
>     while(1);
> }
> 
>  mm/slub.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index 1bb2a93cf7..3eb2e7d386 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3522,7 +3522,18 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,
>  	}
>  
>  	slub_put_cpu_ptr(s->cpu_slab);
> +	if (node != NUMA_NO_NODE) {
> +		slab = new_slab(s, gfpflags | __GFP_THISNODE, node);
> +		if (slab)
> +			goto slab_alloced;
> +
> +		slab = get_any_partial(s, &pc);
> +		if (slab)
> +			goto slab_alloced;
> +	}
>  	slab = new_slab(s, gfpflags, node);
> +
> +slab_alloced:
>  	c = slub_get_cpu_ptr(s->cpu_slab);
>  
>  	if (unlikely(!slab)) {