Re: mlx4: panic during shutdown

From: Tariq Toukan
Date: Thu Oct 20 2016 - 11:37:47 EST


Hi Sebastian,

Thanks for the report.
We've encountered this as well, and trying to find the correct way of solving it.

On 19/10/2016 5:35 PM, Sebastian Ott wrote:
Hi,

After a userspace update (fedora 23->24) I reproducibly run into the
following oops during shutdown (on s390):

[ 71.054832] Unable to handle kernel pointer dereference in virtual kernel address space
[ 71.054835] Failing address: 6b6b6b6b6b6b6000 TEID: 6b6b6b6b6b6b6803
[ 71.054838] Fault in home space mode while using kernel ASCE.
[ 71.054847] AS:0000000000f70007 R3:0000000000000024
[ 71.054883] Oops: 0038 ilc:3 [#1] PREEMPT SMP
[ 71.054887] Modules linked in: mlx4_ib ib_core mlx4_en ptp pps_core mlx4_core [...]
[ 71.054912] CPU: 8 PID: 809 Comm: kworker/8:6 Not tainted 4.8.0-02896-g7137af2-dirty #6
[ 71.054913] Hardware name: IBM 2964 N96 704 (LPAR)
[ 71.054919] Workqueue: events linkwatch_event
[ 71.054921] task: 00000000dbea0008 task.stack: 00000000dbea4000
[ 71.054923] Krnl PSW : 0704e00180000000 000003ff8007a496 (mlx4_en_get_phys_port_id+0x66/0xb0 [mlx4_en])
[ 71.054933] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:2 PM:0 RI:0 EA:3
Krnl GPRS: 0000000000000080 0000000000000268 000000000000004e 00000000001c33e0
[ 71.054937] 000003ff8007a486 0000000000882790 6b6b6b6b6b6b6b6b 0000000000000010
[ 71.054939] 00000000dbea7b18 6b6b6b6b6b6b6b6b 00000000dbea7b18 00000000e72e0000
[ 71.054941] 00000000f15ec900 0000000000000000 000003ff8007a486 00000000dbea79c8
[ 71.054950] Krnl Code: 000003ff8007a486: e310b81c0d14 lgf %r1,55324(%r11)
000003ff8007a48c: a71b004b aghi %r1,75
#000003ff8007a490: eb110003000d sllg %r1,%r1,3
>000003ff8007a496: e31190000002 ltg %r1,0(%r1,%r9)
000003ff8007a49c: a7840015 brc 8,3ff8007a4c6
000003ff8007a4a0: 9208a020 mvi 32(%r10),8
000003ff8007a4a4: 4130a007 la %r3,7(%r10)
000003ff8007a4a8: a7290008 lghi %r2,8
[ 71.054965] Call Trace:
[ 71.054969] ([<000003ff8007a486>] mlx4_en_get_phys_port_id+0x56/0xb0 [mlx4_en])
[ 71.054971] ([<0000000000760b94>] rtnl_fill_ifinfo+0x4ec/0xc90)
[ 71.054974] ([<0000000000764fae>] rtmsg_ifinfo_build_skb+0x96/0xe8)
[ 71.054976] ([<0000000000765038>] rtmsg_ifinfo+0x38/0x78)
[ 71.054978] ([<000000000074150e>] netdev_state_change+0x5e/0x70)
[ 71.054981] ([<0000000000765ca6>] linkwatch_do_dev+0x66/0xc8)
[ 71.054983] ([<0000000000765fd6>] __linkwatch_run_queue+0x13e/0x190)
[ 71.054985] ([<0000000000766070>] linkwatch_event+0x48/0x58)
[ 71.054988] ([<0000000000162a2e>] process_one_work+0x3fe/0x820)
[ 71.054990] ([<00000000001630e6>] worker_thread+0x296/0x460)
[ 71.054992] ([<000000000016b41a>] kthread+0x112/0x120)
[ 71.054996] ([<00000000008762b2>] kernel_thread_starter+0x6/0xc)
[ 71.054998] ([<00000000008762ac>] kernel_thread_starter+0x0/0xc)
[ 71.055000] INFO: lockdep is turned off.
[ 71.055001] Last Breaking-Event-Address:
[ 71.055004] [<0000000000294480>] printk+0xc8/0xd0
[ 71.055006]
[ 71.055008] Kernel panic - not syncing: Fatal exception: panic_on_oops


This was observed with 4.8 but it's also reproducible on 4.9-rc1.
In mlx4_en_get_phys_port_id (which looks like it's called from userspace
via sysfs) the data behind mlx4_en_priv->mdev is already freed.

The problem probably is that the lifetime of mlx4_en_priv->mdev seems to
be shorter than that of struct net_device (and mlx4_en_get_phys_port_id
can be called as long as struct net_device exists).
Right. This happens because we've already freed some resources.
One possible solution is to add a check of netif_device_present in dev_get_phys_port_id.

Something like this:

--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -6601,6 +6601,8 @@ int dev_get_phys_port_id(struct net_device *dev,

if (!ops->ndo_get_phys_port_id)
return -EOPNOTSUPP;
+ if (!netif_device_present(dev))
+ return -ENODEV;
return ops->ndo_get_phys_port_id(dev, ppid);
}
EXPORT_SYMBOL(dev_get_phys_port_id);


However, this causes other issues when combining with MTU change.
In MTU change, netif_device_present returns false for a while, causing an unexpected failure of dev_get_phys_port_id.

Regards,
Sebastian

Regards,
Tariq Toukan