ARM/gic-v4: deadlock occurred

[Heyi Guo]

Hi folks,

We observed deadlocks after enabling GICv4 and PCI passthrough on ARM64 virtual
machines, when not pinning VCPU to physical CPU.

We observed below warnings after enabling lockdep debug in kernel:

[  362.847021] =====================================================
[  362.855643] WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
[  362.864840] 4.19.34+ #7 Tainted: G        W
[  362.872314] -----------------------------------------------------
[  362.881034] CPU 0/KVM/51468 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
[  362.890504] 00000000659c1dc9 (fs_reclaim){+.+.}, at: fs_reclaim_acquire.part.22+0x0/0x48
[  362.901413]
[  362.901413] and this task is already holding:
[  362.912976] 000000007318873f (&dev->event_map.vlpi_lock){....}, at: its_irq_set_vcpu_affinity+0x134/0x638
[  362.928626] which would create a new lock dependency:
[  362.936837]  (&dev->event_map.vlpi_lock){....} -> (fs_reclaim){+.+.}
[  362.946449]
[  362.946449] but this new dependency connects a HARDIRQ-irq-safe lock:
[  362.960877]  (&irq_desc_lock_class){-.-.}
[  362.960880]
[  362.960880] ... which became HARDIRQ-irq-safe at:
[  362.981234]   lock_acquire+0xf0/0x258
[  362.988337]   _raw_spin_lock+0x54/0x90
[  362.995543]   handle_fasteoi_irq+0x2c/0x198
[  363.003205]   generic_handle_irq+0x34/0x50
[  363.010787]   __handle_domain_irq+0x68/0xc0
[  363.018500]   gic_handle_irq+0xf4/0x1e0
[  363.025913]   el1_irq+0xc8/0x180
[  363.032683]   _raw_spin_unlock_irq+0x40/0x60
[  363.040512]   finish_task_switch+0x98/0x258
[  363.048254]   __schedule+0x350/0xca8
[  363.055359]   schedule+0x40/0xa8
[  363.062098]   worker_thread+0xd8/0x410
[  363.069340]   kthread+0x134/0x138
[  363.076070]   ret_from_fork+0x10/0x18
[  363.083111]
[  363.083111] to a HARDIRQ-irq-unsafe lock:
[  363.095213]  (fs_reclaim){+.+.}
[  363.095216]
[  363.095216] ... which became HARDIRQ-irq-unsafe at:
[  363.114527] ...
[  363.114530]   lock_acquire+0xf0/0x258
[  363.126269]   fs_reclaim_acquire.part.22+0x3c/0x48
[  363.134206]   fs_reclaim_acquire+0x2c/0x38
[  363.141363]   kmem_cache_alloc_trace+0x44/0x368
[  363.148892]   acpi_os_map_iomem+0x9c/0x208
[  363.155934]   acpi_os_map_memory+0x28/0x38
[  363.162831]   acpi_tb_acquire_table+0x58/0x8c
[  363.170021]   acpi_tb_validate_table+0x34/0x58
[  363.177162]   acpi_tb_get_table+0x4c/0x90
[  363.183741]   acpi_get_table+0x94/0xc4
[  363.190020]   find_acpi_cpu_topology_tag+0x54/0x240
[  363.197404]   find_acpi_cpu_topology_package+0x28/0x38
[  363.204985]   init_cpu_topology+0xdc/0x1e4
[  363.211498]   smp_prepare_cpus+0x2c/0x108
[  363.217882]   kernel_init_freeable+0x130/0x508
[  363.224699]   kernel_init+0x18/0x118
[  363.230624]   ret_from_fork+0x10/0x18
[  363.236611]
[  363.236611] other info that might help us debug this:
[  363.236611]
[  363.251604] Chain exists of:
[  363.251604]   &irq_desc_lock_class --> &dev->event_map.vlpi_lock --> fs_reclaim
[  363.251604]
[  363.270508]  Possible interrupt unsafe locking scenario:
[  363.270508]
[  363.282238]        CPU0                    CPU1
[  363.289228]        ----                    ----
[  363.296189]   lock(fs_reclaim);
[  363.301726]                                local_irq_disable();
[  363.310122] lock(&irq_desc_lock_class);
[  363.319143] lock(&dev->event_map.vlpi_lock);
[  363.328617]   <Interrupt>
[  363.333713]     lock(&irq_desc_lock_class);
[  363.340414]
[  363.340414]  *** DEADLOCK ***
[  363.340414]
[  363.353682] 5 locks held by CPU 0/KVM/51468:
[  363.360412]  #0: 00000000eeb852a5 (&vdev->igate){+.+.}, at: vfio_pci_ioctl+0x2f8/0xed0
[  363.370915]  #1: 000000002ab491f7 (lock#9){+.+.}, at: irq_bypass_register_producer+0x6c/0x1d0
[  363.382139]  #2: 000000000d9fd5c6 (&its->its_lock){+.+.}, at: kvm_vgic_v4_set_forwarding+0xd0/0x188
[  363.396625]  #3: 00000000232bdc47 (&irq_desc_lock_class){-.-.}, at: __irq_get_desc_lock+0x60/0xa0
[  363.408486]  #4: 000000007318873f (&dev->event_map.vlpi_lock){....}, at: its_irq_set_vcpu_affinity+0x134/0x638


Then we found that irq_set_vcpu_affinity() in kernel/irq/manage.c aquires an
antomic context by irq_get_desc_lock() at the beginning, but in
its_irq_set_vcpu_affinity() (drivers/irqchip/irq-gic-v3-its.c) we are still
using mutext_lock, kcalloc, kfree, etc, which we think should be forbidden in
atomic context.

Though the issue is observed in 4.19.34, we don't find any related fixes in the mainline yet.

Please advise.

Thanks,

Heyi