Re: [REGRESSION] mlx5: Driver remove during hot unplug is broken

[Niklas Schnelle]

Hi Parav, Hi Shay,

On 7/8/20 12:43 PM, Parav Pandit wrote:
> Hi Niklas,
> 
... snip ...
>>>
> 
> Sorry for my late response.
> Yes, this looks good and I also found same in my analysis.
> With latest code mlx5_pci_close() already does drain_health_wq(), so the additional call in remove_one() is redundant.
> It should be just removed.
> If you can verify below hunk in your setup, it will be really helpful.
> You still need patch 42ea9f1b5c6 in your tree.
> 
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> index 8b658908f044..ebec2318dbc4 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
> @@ -1382,7 +1382,6 @@ static void remove_one(struct pci_dev *pdev)
> 
>         devlink_reload_disable(devlink);
>         mlx5_crdump_disable(dev);
> -       mlx5_drain_health_wq(dev);
>         mlx5_unload_one(dev, true);
>         mlx5_pci_close(dev);
>         mlx5_mdev_uninit(dev);
> 

thanks for looking into this. Sadly it looks like this
doesn't quite work, I'm getting the bewlow illegal dereference after
removing the mlx5_drain_health_wq(dev) from remove_one() on top of
v5.8-rc4.
I think this is similar to what happned when I tried recreating the
reordering on -rc3 which is why I contacted Shay as I wasn't
able to get this to work again with the current code.
(Sorry if there are formatting issues with the below, plugged this
out of a x3270).

64.735421Â Unable to handle kernel pointer dereference in virtual kernel address space 
64.735470Â Failing address: 6b6b6b6b6b6b6000 TEID: 6b6b6b6b6b6b6803 
64.735476Â Fault in home space mode while using kernel ASCE. 
64.735484Â AS:0000000088da0007 R3:0000000000000024  
64.735533Â Oops: 0038 ilc:3 Ã#1Â PREEMPT SMP  
64.735538Â Modules linked in: rpcrdma sunrpc rdma_ucm rdma_cm iw_cm ib_cm configfs mlx5_ib ib_uverbs ib_core mlx5_core dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua s390_trng ghash_s390 prng ctr aes_s390 des_s390 libdes sha3_512_s390 sha3_256_s390 sha512_s390 sha1_s390 vfio_ccw vfi
64.735421Â Unable to handle kernel pointer dereference in virtual kernel address space 
64.735558Â CPU: 0 PID: 762 Comm: kworker/u128:3 Not tainted 5.8.0-rc4-dirty #2 
64.735561Â Hardware name: IBM 3906 M04 704 (z/VM 7.1.0) 
64.735618Â Workqueue: mlx5_health0000:00:00.0 mlx5_fw_fatal_reporter_err_work mlx5_core
64.735623Â Krnl PSW : 0704e00180000000 00000000876936c6 (notifier_call_chain+0x3e/0xe8)01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 01.01: HCPGSP2629I The virtual machine is placed in CP mode due to a SIGP stop from CPU 00.
64.735702Â            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3 
64.735704Â Krnl GPRS: 00000000a38fc246 0000000080000001 00000000e814db00 0000000000000080 
64.735706Â            0000000000000001 ffffffffffffffff 0000000000000000 0000000000000001 
64.735708Â            0000000000000080 0000000000000001 ffffffffffffffff 6b6b6b6b6b6b6b6b 
64.735710Â            00000000e5ff0100 0000000000000000 000003e000973bf8 000003e000973ba0 
64.735716Â Krnl Code: 00000000876936ba: b9040083            lgr     %r8,%r3 
64.735716Â            00000000876936be: b9040074            lgr     %r7,%r4 
64.735716Â           #00000000876936c2: a7d80000            lhi     %r13,0 
64.735716Â           >00000000876936c6: e390b0080004        lg      %r9,8(%r11) 
64.735716Â            00000000876936cc: e320b0000004        lg      %r2,0(%r11) 
64.735716Â            00000000876936d2: c0e5ffffd773        brasl   %r14,000000008768e5b8 
64.735716Â            00000000876936d8: ec280032007e        cij     %r2,0,8,000000008769373c 
64.735716Â            00000000876936de: e310b0000004        lg      %r1,0(%r11) 
64.735731Â Call Trace: 
64.735740Â  <00000000876936c6>Â notifier_call_chain+0x3e/0xe8  
64.735764Â  <000000008769392c>Â __atomic_notifier_call_chain+0x9c/0x138  
64.735766Â  <00000000876939f2>Â atomic_notifier_call_chain+0x2a/0x38  
64.735782Â  <000003ff801e99c4>Â mlx5_enter_error_state+0xec/0x100 mlx5_core
64.735797  <000003ff801e9a0c> mlx5_fw_fatal_reporter_err_work+0x34/0xb8 mlx5_core  
64.735802Â  <000000008768784c>Â process_one_work+0x27c/0x478  
64.735805Â  <0000000087687aae>Â worker_thread+0x66/0x368  
64.735807Â  <0000000087690a96>Â kthread+0x176/0x1a0  
64.735811Â  <0000000088202820>Â ret_from_fork+0x24/0x2c  
64.735812Â INFO: lockdep is turned off. 
64.735814Â Last Breaking-Event-Address: 
64.735816Â  <0000000087693926>Â __atomic_notifier_call_chain+0x96/0x138 
64.735820Â Kernel panic - not syncing: Fatal exception: panic_on_oops 
00: HCPGIR450W CP entered; disabled wait PSW 00020001 80000000 00000000 8762103E