Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 13.02.26 22:35, mhklkml@xxxxxxxxxxxx wrote:
> From: Jan Kiszka <jan.kiszka@xxxxxxxxxxx> Sent: Thursday, February 12, 2026 8:06 AM
>>
>> On 09.02.26 19:25, Michael Kelley wrote:
>>> From: Florian Bezdeka <florian.bezdeka@xxxxxxxxxxx> Sent: Monday, February 9, 2026 2:35 AM
>>>>
>>>> On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:
>>>>
>>>> [snip]
>>>>>
>>>>> I've run your suggested experiment on an arm64 VM in the Azure cloud. My
>>>>> kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
>>>>> CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
>>>>> (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
>>>>> The VM comes up and runs, but with this warning during boot:
>>>>>
>>>>> [    3.075604] hv_utils: Registering HyperV Utility Driver
>>>>> [    3.075636] hv_vmbus: registering driver hv_utils
>>>>> [    3.085920] =============================
>>>>> [    3.088128] hv_vmbus: registering driver hv_netvsc
>>>>> [    3.091180] [ BUG: Invalid wait context ]
>>>>> [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
>>>>> [    3.097582] -----------------------------
>>>>> [    3.099899] systemd-udevd/284 is trying to lock:
>>>>> [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
>>>>> [    3.108208] other info that might help us debug this:
>>>>> [    3.111454] context-{2:2}
>>>>> [    3.112987] 1 lock held by systemd-udevd/284:
>>>>> [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
>>>>> [    3.121224] stack backtrace:
>>>>> [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E 6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
>>>>> [    3.129631] Tainted: [E]=UNSIGNED_MODULE
>>>>> [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
>>>>> [    3.138553] Call trace:
>>>>> [    3.140015]  show_stack+0x20/0x38 (C)
>>>>> [    3.142137]  dump_stack_lvl+0x9c/0x158
>>>>> [    3.144340]  dump_stack+0x18/0x28
>>>>> [    3.146290]  __lock_acquire+0x488/0x1e20
>>>>> [    3.148569]  lock_acquire+0x11c/0x388
>>>>> [    3.150703]  rt_spin_lock+0x54/0x230
>>>>> [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
>>>>> [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
>>>>> [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
>>>>> [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
>>>>> [    3.163495]  handle_irq_desc+0x48/0x68
>>>>> [    3.165851]  generic_handle_domain_irq+0x20/0x38
>>>>> [    3.168664]  gic_handle_irq+0x1dc/0x430
>>>>> [    3.170868]  call_on_irq_stack+0x30/0x70
>>>>> [    3.173161]  do_interrupt_handler+0x88/0xa0
>>>>> [    3.175724]  el1_interrupt+0x4c/0xb0
>>>>> [    3.177855]  el1h_64_irq_handler+0x18/0x28
>>>>> [    3.180332]  el1h_64_irq+0x84/0x88
>>>>> [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
>>>>> [    3.185493]  rt_mutex_slowunlock+0x404/0x440
>>>>> [    3.187951]  rt_spin_unlock+0xb8/0x178
>>>>> [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
>>>>> [    3.193100]  alloc_empty_file+0x64/0x148
>>>>> [    3.195461]  path_openat+0x58/0xaa0
>>>>> [    3.197658]  do_file_open+0xa0/0x140
>>>>> [    3.199752]  do_sys_openat2+0x190/0x278
>>>>> [    3.202124]  do_sys_open+0x60/0xb8
>>>>> [    3.204047]  __arm64_sys_openat+0x2c/0x48
>>>>> [    3.206433]  invoke_syscall+0x6c/0xf8
>>>>> [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
>>>>> [    3.211050]  do_el0_svc+0x24/0x38
>>>>> [    3.212990]  el0_svc+0x164/0x3c8
>>>>> [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
>>>>> [    3.217251]  el0t_64_sync+0x1b0/0x1b8
>>>>> [    3.219450] hv_utils: Heartbeat IC version 3.0
>>>>> [    3.219471] hv_utils: Shutdown IC version 3.2
>>>>> [    3.219844] hv_utils: TimeSync IC version 4.0
>>>>
>>>> That matches with my expectation that the same problem exists on arm64.
>>>> The patch from Jan addresses that issue for x86 (only, so far) as we do
>>>> not have a working test environment for arm64 yet.
>>>
>>> OK. I had understood Jan's earlier comments to mean that the VMBus
>>> interrupt problem was implicitly solved on arm64 because of VMBus using
>>> a standard Linux IRQ on arm64. But evidently that's not the case. So my
>>> earlier comment stands: The code changes should go into the architecture
>>> independent portion of the VMBus driver, and not under arch/x86. I
>>> can probably work with you to test on arm64 if need be.
>>>
>>
>> I can move the code, sure, but I still haven't understood what
>> invalidates my assumptions (beside what you observed). vmbus_drv calls
>> request_percpu_irq, and that is - as far as I can see - not injecting
>> IRQF_NO_THREAD. Any explanations welcome.
> 
> I haven't setup detailed debugging on arm64 yet, but in prep for that
> I went looking at the places in the kernel IRQ handling where
> IRQF_NO_THREAD influences behavior. The key function appears to be
> irq_setup_forced_threading(). This function first checks force_irqthreads(),
> which will be "true" when PREEMPT_RT is set. The function then checks
> the IRQF_NO_THREAD flag and the IRQF_PERCPU flag. From what I can
> see, the IRQF_PERCPU flag is treated like the IRQF_NO_THREAD flag, and
> causes forced threading to *not* be done. So the behavior ends up being
> the same as when PREEMPT_RT is not set.
> 
> Since the VMBus interrupt is a per-cpu interrupt, forced threading is not
> done. In that case, the stack trace I reported makes sense. Take a look at
> the code and see if you agree.

Indeed, missed the IRQF_PERCPU impact on auto-threading. I'll rework my
patch to perform the transition arch-independently.

Thanks,
Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center