Lockdep warning in iwl_pcie_rx_handle()

From: Jiri Kosina
Date: Tue Mar 02 2021 - 02:09:14 EST

Hi,

I am getting the splat below with Linus' tree as of today (5.11-rc1,
fe07bfda2fb). I haven't started to look into the code yet, but apparently
this has been already reported by Heiner here:

https://www.spinics.net/lists/linux-wireless/msg208353.html

so before I start digging deep into it (the previous kernel this
particular machine had is 5.9, so I'd rather avoid lenghty bisect for now
in case someone has already looked into it and has ideas where the problem
is), I thought I'd ask whether this has been root-caused elsewhere
already.

Thanks.




[ 9.970414] ========================================================
[ 9.973157] WARNING: possible irq lock inversion dependency detected
[ 9.976186] 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
[ 9.979239] --------------------------------------------------------
[ 9.982223] irq/136-iwlwifi/565 just changed the state of lock:
[ 9.984904] ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 9.987932] but this lock took another, SOFTIRQ-unsafe lock in the past:
[ 9.990980] (napi_hash_lock){+.+.}-{2:2}
[ 9.990986]

and interrupts could create inverse lock ordering between them.

[ 10.003955]
other info that might help us debug this:
[ 10.008989] Possible interrupt unsafe locking scenario:

[ 10.013966] CPU0 CPU1
[ 10.016404] ---- ----
[ 10.018976] lock(napi_hash_lock);
[ 10.021529] local_irq_disable();
[ 10.023985] lock(&rxq->lock);
[ 10.026340] lock(napi_hash_lock);
[ 10.028644] <Interrupt>
[ 10.030934] lock(&rxq->lock);
[ 10.033119]
*** DEADLOCK ***

[ 10.039456] 1 lock held by irq/136-iwlwifi/565:
[ 10.041549] #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30 [iwlwifi]
[ 10.043728]
the shortest dependencies between 2nd lock and 1st lock:
[ 10.047836] -> (napi_hash_lock){+.+.}-{2:2} {
[ 10.049845] HARDIRQ-ON-W at:
[ 10.051860] lock_acquire+0x277/0x3d0
[ 10.053881] _raw_spin_lock+0x2c/0x40
[ 10.056170] netif_napi_add+0x14b/0x270
[ 10.058349] e1000_probe+0x2fe/0xee0 [e1000e]
[ 10.060539] local_pci_probe+0x42/0x90
[ 10.062609] pci_device_probe+0x10b/0x1c0
[ 10.064579] really_probe+0xef/0x4b0
[ 10.066578] driver_probe_device+0xde/0x150
[ 10.068765] device_driver_attach+0x4f/0x60
[ 10.070890] __driver_attach+0x9c/0x140
[ 10.073115] bus_for_each_dev+0x79/0xc0
[ 10.075304] bus_add_driver+0x18d/0x220
[ 10.077464] driver_register+0x5b/0xf0
[ 10.079615] do_one_initcall+0x5b/0x300
[ 10.081686] do_init_module+0x5b/0x21c
[ 10.083730] load_module+0x1dae/0x22c0
[ 10.085751] __do_sys_finit_module+0xad/0x110
[ 10.087799] do_syscall_64+0x33/0x80
[ 10.089758] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.091747] SOFTIRQ-ON-W at:
[ 10.093634] lock_acquire+0x277/0x3d0
[ 10.095514] _raw_spin_lock+0x2c/0x40
[ 10.097353] netif_napi_add+0x14b/0x270
[ 10.099208] e1000_probe+0x2fe/0xee0 [e1000e]
[ 10.101032] local_pci_probe+0x42/0x90
[ 10.102835] pci_device_probe+0x10b/0x1c0
[ 10.104648] really_probe+0xef/0x4b0
[ 10.106439] driver_probe_device+0xde/0x150
[ 10.106443] device_driver_attach+0x4f/0x60
[ 10.106446] __driver_attach+0x9c/0x140
[ 10.111451] bus_for_each_dev+0x79/0xc0
[ 10.111454] bus_add_driver+0x18d/0x220
[ 10.111456] driver_register+0x5b/0xf0
[ 10.111459] do_one_initcall+0x5b/0x300
[ 10.111462] do_init_module+0x5b/0x21c
[ 10.111467] load_module+0x1dae/0x22c0
[ 10.111469] __do_sys_finit_module+0xad/0x110
[ 10.122761] do_syscall_64+0x33/0x80
[ 10.122768] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.122772] INITIAL USE at:
[ 10.122775] lock_acquire+0x277/0x3d0
[ 10.128672] _raw_spin_lock+0x2c/0x40
[ 10.130160] netif_napi_add+0x14b/0x270
[ 10.131725] e1000_probe+0x2fe/0xee0 [e1000e]
[ 10.133268] local_pci_probe+0x42/0x90
[ 10.134783] pci_device_probe+0x10b/0x1c0
[ 10.136325] really_probe+0xef/0x4b0
[ 10.137864] driver_probe_device+0xde/0x150
[ 10.139394] device_driver_attach+0x4f/0x60
[ 10.140908] __driver_attach+0x9c/0x140
[ 10.142429] bus_for_each_dev+0x79/0xc0
[ 10.143947] bus_add_driver+0x18d/0x220
[ 10.145490] driver_register+0x5b/0xf0
[ 10.147024] do_one_initcall+0x5b/0x300
[ 10.148566] do_init_module+0x5b/0x21c
[ 10.150013] load_module+0x1dae/0x22c0
[ 10.151293] __do_sys_finit_module+0xad/0x110
[ 10.152571] do_syscall_64+0x33/0x80
[ 10.153740] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.154889] }
[ 10.156023] ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
[ 10.157191] ... acquired at:
[ 10.158318] _raw_spin_lock+0x2c/0x40
[ 10.159418] netif_napi_add+0x14b/0x270
[ 10.160500] _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
[ 10.161609] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
[ 10.162704] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
[ 10.163813] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
[ 10.164955] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
[ 10.166079] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
[ 10.167206] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
[ 10.168326] iwl_opmode_register+0x71/0xe0 [iwlwifi]
[ 10.169451] iwl_mvm_init+0x34/0x1000 [iwlmvm]
[ 10.170585] do_one_initcall+0x5b/0x300
[ 10.171699] do_init_module+0x5b/0x21c
[ 10.172818] load_module+0x1dae/0x22c0
[ 10.173933] __do_sys_finit_module+0xad/0x110
[ 10.175048] do_syscall_64+0x33/0x80
[ 10.176164] entry_SYSCALL_64_after_hwframe+0x44/0xae

[ 10.178421] -> (&rxq->lock){+.-.}-{2:2} {
[ 10.179564] HARDIRQ-ON-W at:
[ 10.180709] lock_acquire+0x277/0x3d0
[ 10.181874] _raw_spin_lock_bh+0x34/0x40
[ 10.183049] _iwl_pcie_rx_init+0x130/0x710 [iwlwifi]
[ 10.184229] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
[ 10.185397] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
[ 10.186572] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
[ 10.187759] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
[ 10.188950] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
[ 10.190132] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
[ 10.191312] iwl_opmode_register+0x71/0xe0 [iwlwifi]
[ 10.192497] iwl_mvm_init+0x34/0x1000 [iwlmvm]
[ 10.193685] do_one_initcall+0x5b/0x300
[ 10.194868] do_init_module+0x5b/0x21c
[ 10.196048] load_module+0x1dae/0x22c0
[ 10.197225] __do_sys_finit_module+0xad/0x110
[ 10.198409] do_syscall_64+0x33/0x80
[ 10.199601] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.200801] IN-SOFTIRQ-W at:
[ 10.202000] lock_acquire+0x277/0x3d0
[ 10.203202] _raw_spin_lock+0x2c/0x40
[ 10.204412] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.205653] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
[ 10.206879] __napi_poll+0x2d/0x220
[ 10.208075] net_rx_action+0x106/0x280
[ 10.209276] __do_softirq+0xd4/0x461
[ 10.210486] do_softirq+0x63/0xa0
[ 10.211677] __local_bh_enable_ip+0xd1/0xf0
[ 10.212879] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
[ 10.214092] irq_thread_fn+0x21/0x60
[ 10.215298] irq_thread+0x107/0x1a0
[ 10.216491] kthread+0x13e/0x160
[ 10.217670] ret_from_fork+0x22/0x30
[ 10.218855] INITIAL USE at:
[ 10.220030] lock_acquire+0x277/0x3d0
[ 10.221239] _raw_spin_lock_bh+0x34/0x40
[ 10.222438] _iwl_pcie_rx_init+0x130/0x710 [iwlwifi]
[ 10.223667] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
[ 10.224894] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
[ 10.226167] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
[ 10.227448] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
[ 10.228718] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
[ 10.229977] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
[ 10.231236] iwl_opmode_register+0x71/0xe0 [iwlwifi]
[ 10.232492] iwl_mvm_init+0x34/0x1000 [iwlmvm]
[ 10.233758] do_one_initcall+0x5b/0x300
[ 10.235002] do_init_module+0x5b/0x21c
[ 10.236224] load_module+0x1dae/0x22c0
[ 10.237454] __do_sys_finit_module+0xad/0x110
[ 10.238679] do_syscall_64+0x33/0x80
[ 10.239906] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.241137] }
[ 10.242327] ... key at: [<ffffffffc111f0c0>] __key.69825+0x0/0xfffffffffffe2f40 [iwlwifi]
[ 10.243587] ... acquired at:
[ 10.244814] __lock_acquire+0x27d/0x1a50
[ 10.246055] lock_acquire+0x277/0x3d0
[ 10.247296] _raw_spin_lock+0x2c/0x40
[ 10.248518] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.249789] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
[ 10.251028] __napi_poll+0x2d/0x220
[ 10.252254] net_rx_action+0x106/0x280
[ 10.253484] __do_softirq+0xd4/0x461
[ 10.254731] do_softirq+0x63/0xa0
[ 10.255946] __local_bh_enable_ip+0xd1/0xf0
[ 10.257157] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
[ 10.258382] irq_thread_fn+0x21/0x60
[ 10.259608] irq_thread+0x107/0x1a0
[ 10.260819] kthread+0x13e/0x160
[ 10.262022] ret_from_fork+0x22/0x30

[ 10.264433]
stack backtrace:
[ 10.266755] CPU: 2 PID: 565 Comm: irq/136-iwlwifi Not tainted 5.12.0-rc1-00002-gbada49429032 #5
[ 10.267964] Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
[ 10.269175] Call Trace:
[ 10.270376] <IRQ>
[ 10.271566] dump_stack+0x7f/0xa1
[ 10.272750] mark_lock.part.49+0x8cb/0xc70
[ 10.273950] ? __lock_acquire+0x2e1/0x1a50
[ 10.275164] ? __lock_acquire+0x27d/0x1a50
[ 10.276372] __lock_acquire+0x27d/0x1a50
[ 10.277562] ? __lock_acquire+0x2e1/0x1a50
[ 10.278754] ? __lock_acquire+0x2e1/0x1a50
[ 10.279937] lock_acquire+0x277/0x3d0
[ 10.281110] ? iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.282317] ? lock_is_held_type+0xb4/0x120
[ 10.283483] _raw_spin_lock+0x2c/0x40
[ 10.284637] ? iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.285813] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.286985] ? __lock_acquire+0x2e1/0x1a50
[ 10.288148] ? try_to_wake_up+0x41c/0x600
[ 10.289323] ? print_dl_rq+0xcc/0x160
[ 10.290490] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
[ 10.291684] __napi_poll+0x2d/0x220
[ 10.292862] net_rx_action+0x106/0x280
[ 10.294034] __do_softirq+0xd4/0x461
[ 10.295185] do_softirq+0x63/0xa0
[ 10.296349] </IRQ>
[ 10.297514] __local_bh_enable_ip+0xd1/0xf0
[ 10.298683] ? iwl_pcie_irq_handler+0x66c/0xb30 [iwlwifi]
[ 10.299882] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
[ 10.301076] ? irq_forced_thread_fn+0x80/0x80
[ 10.302265] irq_thread_fn+0x21/0x60
[ 10.303434] ? irq_thread+0xa1/0x1a0
[ 10.304596] irq_thread+0x107/0x1a0
[ 10.305766] ? wake_threads_waitq+0x30/0x30
[ 10.306922] ? irq_thread_dtor+0xa0/0xa0
[ 10.308078] kthread+0x13e/0x160
[ 10.309246] ? kthread_create_worker_on_cpu+0x70/0x70
[ 10.310410] ret_from_fork+0x22/0x30