Re: [PATCH v9 21/24] x86/resctrl: Allow overflow/limbo handlers to be scheduled on any-but cpu

[Tony Luck]

> -static inline unsigned int cpumask_any_housekeeping(const struct cpumask *mask)
> +static inline unsigned int
> +cpumask_any_housekeeping(const struct cpumask *mask, int exclude_cpu)
>  {
>  	unsigned int cpu, hk_cpu;
>  
> -	cpu = cpumask_any(mask);
> -	if (!tick_nohz_full_cpu(cpu))
> +	if (exclude_cpu == RESCTRL_PICK_ANY_CPU)
> +		cpu = cpumask_any(mask);
> +	else
> +		cpu = cpumask_any_but(mask, exclude_cpu);
> +
> +	if (!IS_ENABLED(CONFIG_NO_HZ_FULL))
> +		return cpu;
> +
> +	/* If the CPU picked isn't marked nohz_full nothing more needs doing. */
> +	if (cpu < nr_cpu_ids && !tick_nohz_full_cpu(cpu))
>  		return cpu;
>  
> +	/* Try to find a CPU that isn't nohz_full to use in preference */
>  	hk_cpu = cpumask_nth_andnot(0, mask, tick_nohz_full_mask);
> +	if (hk_cpu == exclude_cpu)
> +		hk_cpu = cpumask_nth_andnot(1, mask, tick_nohz_full_mask);

               OOPS HERE ^^^^^^^^^^^^^^^^^^
> +
>  	if (hk_cpu < nr_cpu_ids)
>  		cpu = hk_cpu;

James,

I'm playing around with v6.9-rc1. When I take all the CPUs in a domain
offline, resctrl wants to stop the mbm overflow handler. While doing
so it crashes with a NULL dereference in the code above.

At this point the cpu_mask for the domain still has the bit for
the last CPU standing in this domain. So cpumask_any_housekeeping()
is called with a mask that just has CPU 36 set, but request to find
a CPU other than 36.

-Tony

Heres the OOPs:

[   97.107355] smpboot: CPU 37 is now offline
[   97.166136] BUG: kernel NULL pointer dereference, address: 0000000000000000
[   97.173118] #PF: supervisor read access in kernel mode
[   97.178263] #PF: error_code(0x0000) - not-present page
[   97.183410] PGD 0
[   97.185438] Oops: 0000 [#1] PREEMPT SMP NOPTI
[   97.189805] CPU: 36 PID: 235 Comm: cpuhp/36 Tainted: G                T  6.9.0-rc1 #356
[   97.197810] Hardware name: Intel Corporation WilsonCity/WilsonCity, BIOS WLYDCRB1.SYS.0021.P06.2104260458 04/26/2021
[   97.208322] RIP: 0010:__find_nth_andnot_bit+0x66/0x110
[   97.213472] Code: d2 eb 20 48 29 c6 48 8d 41 c0 48 39 ca 0f 82 8b 00 00 00 48 01 f0 48 83 c1 40 48 39 d0 0f 83 9e 00 00 00 4d 89 d3 49 83 c2 01 <4b> 8b 3c d8 48 f7 d7 4b 23 3c d9 f3 48 0f b8 c7 48 39 c6 73 c4 4c
[   97.232214] RSP: 0018:ff462f11872bbd98 EFLAGS: 00010202
[   97.237440] RAX: 0000000000000000 RBX: 0000000000000024 RCX: 0000000000000080
[   97.244574] RDX: 0000000000000090 RSI: 0000000000000000 RDI: 0000000000000000
[   97.251705] RBP: ff43a0b049599018 R08: 0000000000000000 R09: ff43a0b049599018
[   97.258835] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000025
[   97.265970] R13: 0000000000000090 R14: 0000000000000000 R15: ffffffffaa068fb0
[   97.273100] FS:  0000000000000000(0000) GS:ff43a0b81d800000(0000) knlGS:0000000000000000
[   97.281187] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   97.286932] CR2: 0000000000000000 CR3: 0000000a50e3a004 CR4: 0000000000771ef0
[   97.294065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   97.301197] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   97.308328] PKRU: 55555554
[   97.311039] Call Trace:
[   97.313492]  <TASK>
[   97.315600]  ? __die+0x1f/0x60
[   97.318665]  ? page_fault_oops+0x176/0x5a0
[   97.322777]  ? exc_page_fault+0x7f/0x260
[   97.326708]  ? asm_exc_page_fault+0x22/0x30
[   97.330903]  ? __pfx_resctrl_arch_offline_cpu+0x10/0x10
[   97.336140]  ? __find_nth_andnot_bit+0x66/0x110
[   97.340680]  ? __cancel_work+0x7d/0xc0
[   97.344438]  cpumask_any_housekeeping+0x55/0x110
[   97.349060]  mbm_setup_overflow_handler+0x40/0x70
[   97.353774]  resctrl_offline_cpu+0x101/0x110
[   97.358054]  resctrl_arch_offline_cpu+0x19/0x260
[   97.362673]  cpuhp_invoke_callback+0x156/0x6b0
[   97.367127]  ? cpuhp_thread_fun+0x5f/0x250
[   97.371234]  cpuhp_thread_fun+0x1ca/0x250
[   97.375247]  ? __pfx_smpboot_thread_fn+0x10/0x10
[   97.379874]  smpboot_thread_fn+0x184/0x220
[   97.383974]  kthread+0xe0/0x110
[   97.387129]  ? __pfx_kthread+0x10/0x10
[   97.390888]  ret_from_fork+0x2d/0x50
[   97.394467]  ? __pfx_kthread+0x10/0x10
[   97.398220]  ret_from_fork_asm+0x1a/0x30
[   97.402149]  </TASK>