Re: kmemleak: sched_domain_shared leaked on asymmetric-capacity + SCHED_CACHE

[Dietmar Eggemann]

Hi Breno & Prateek,

On 04.07.26 08:16, K Prateek Nayak wrote:
> Hello Breno,
> 
> On 7/3/2026 9:49 PM, Breno Leitao wrote:
>> Hello Prateek,
>>
>> On Fri, Jul 03, 2026 at 04:05:59PM +0530, K Prateek Nayak wrote:
>>> On 7/3/2026 3:52 PM, Breno Leitao wrote:
>>>> On current linux-next (CONFIG_SCHED_CACHE=y) we hit a kmemleak on an
>>>> arm64 box with asymmetric CPU capacity, triggered by a cpuset-driven
>>>> sched-domain rebuild:
>>>>
>>>> kmemleak: unreferenced object 0xffff000100c95e80 (size 32):
>>>>   comm "kworker/22:1", pid 407, jiffies 4294669077
>>>>   hex dump (first 32 bytes):
>>>>     48 00 00 00 48 00 00 00 00 00 00 00 20 00 00 00  H...H....... ...
>>>>   backtrace (crc ec5d7053):
>>>>     __kmalloc_cache_node_noprof
>>>>     build_sched_domains
>>>>     partition_sched_domains
>>>>     rebuild_sched_domains_locked
>>>>     rebuild_sched_domains
>>>>     process_scheduled_works
>>>>     kthread
>>>>     ret_from_fork
>>>>   kmemleak: 1 new suspected memory leaks
>>>>
>>>> The leaked object is a struct sched_domain_shared (32 bytes with
>>>> CONFIG_SCHED_CACHE) allocated in __sds_alloc(), inlined into
>>>> build_sched_domains()
>>>>
>>>> Is this a known issue?
>>>
>>> It is theoretically possible but there is a defensive WARN_ON()
>>> in topology code that you should have hit first. Do you see any
>>> other warning?
>>>
>>> If it is not too much trouble, could you add "sched_verbose"
>>> to your kernel cmdline (or do
>>> echo Y > /sys/kernel/debug/sched/verbose) and redo this cpuset
>>> that leaks the data and share the dmesg. It should give some clue
>>> what the topology looks like that causes this.
>>
>> Sure, does this one help:
> 
> Unfortunately no! This doesn't give the span and capacity
> information so I cannot really tell what kind of topology
> actually triggers this.

I'm trying to recreate this on Arm64 qemu with dtb file to get the
asymmetric CPU capacities (emulating Pixel6):

$ cat /sys/devices/system/cpu/cpu*/cpu_capacity
160
160
160
160
498
498
1024
1024

I'm on linux_next master:

eb8711ece2c - (HEAD -> linux-next, tag: next-20260702, linux-next/master)

so it has:

9e005ed21152 - sched/topology: Allow multiple domains to claim
sched_domain_shared (2026-05-19 K Prateek Nayak)

$ cat /sys/kernel/debug/sched/domains/cpu0/domain0/{name,flags}
MC
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE
SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING

8 CPUs flat:

$ cat /proc/schedstat | grep "^c\|^d" | awk '{ print $1 " " $2 " " $3}'
cpu0 0 0
domain0 MC ff
cpu1 0 0
domain0 MC ff
cpu2 0 0
domain0 MC ff
cpu3 0 0
domain0 MC ff
cpu4 0 0
domain0 MC ff
cpu5 0 0
domain0 MC ff
cpu6 0 0
domain0 MC ff
cpu7 0 0
domain0 MC ff

> Ideally, you enable verbose logging, do the cpuset operation
> that leaks the object, and then save the demesg after which
> will have a complete topology for all CPUs there including
> their capacities and we can piece together a qemu environment
> to debug where it all went wrong.

+1

$ cat /proc/cmdline
... sched_verbose=1

What exactly is this cpuset operation forcing a sched domain rebuild?

I'm trying a split into 2 exclusive cpusets {0,2,4,6} and {1,3,5,7} to
get 2 asymmetric CPU capacity islands.

...

[   95.747359] root domain span: 1,3,5,7
[   95.747760] root domain span: 0,2,4,6
[   95.747771] _sched_cache_active_set: enabling cache aware scheduling
[   95.747788] rd 1,3,5,7: Checking EAS: frequency-invariant load
tracking not yet supported
...

$ cat /sys/kernel/debug/sched/domains/cpu0/domain0/{name,flags}
MC
SD_BALANCE_NEWIDLE SD_BALANCE_EXEC SD_BALANCE_FORK SD_WAKE_AFFINE
SD_ASYM_CPUCAPACITY SD_ASYM_CPUCAPACITY_FULL SD_SHARE_LLC SD_PREFER_SIBLING

$ cat /proc/schedstat | grep "^c\|^d" | awk '{ print $1 " " $2 " " $3}'
cpu0 0 0
domain0 MC 55
cpu1 0 0
domain0 MC aa
cpu2 0 0
domain0 MC 55
cpu3 0 0
domain0 MC aa
cpu4 0 0
domain0 MC 55
cpu5 0 0
domain0 MC aa
cpu6 0 0
domain0 MC 55
cpu7 0 0
domain0 MC aa

But this hasn't triggered kmemleak so far.

That said, it triggered once (~600s uptime) but w/o any cpuset
operation? Never happened again since then :-(

[...]

> Is this the defualt topology before the cpuset? Because
> min_interval of 8 means there are 8 cPUs in the span and
> you have 8 CPUs in your machine so there seems to be a
> single root partition when you captured this.

+1

> Also does your dmesg have any warnings from topology.c?
> 
> The only way I can see this triggering is there is a single
> cpu SD_SHARE_LLC at top which claims all the sd->shared and
> then you get a SD_ASYM_CPUCAPACITY_FULL on top which too
> tries to claim a sd->shared but all are already gone.

?

> Afaict, all CPUs in your system are on single LLC and you
> can't SD_ASYM_CPUCAPACITY_FULL without having more than one
> capacity in the span so I'm at a loss currently as to how
> this reference gets dropped without being freed.

$ cat /sys/devices/system/cpu/cpu*/cpu_capacity  should tell.


Another thing: does your system enable 'cache aware scheduling', i.e. do
you not see:

pr_info("%s: cache aware scheduling not supported on this platform\n",
__func__);