Re: [PATCH] s390: vfio-ap: disable IRQ in remove callback results in kernel OOPS
From: Pierre Morel
Date: Wed Oct 30 2019 - 14:02:26 EST
On 10/30/19 5:51 PM, Tony Krowiak wrote:
On 10/30/19 10:00 AM, Pierre Morel wrote:
On 10/30/19 8:44 AM, Harald Freudenberger wrote:
On 29.10.19 23:09, Tony Krowiak wrote:
From: aekrowia <akrowiak@xxxxxxxxxxxxx>
When an AP adapter card is configured off via the SE or the SCLP
Deconfigure Adjunct Processor command and the AP bus subsequently
detects
that the adapter card is no longer in the AP configuration, the card
device representing the adapter card as well as each of its associated
AP queue devices will be removed by the AP bus. If one or more of the
affected queue devices is bound to the VFIO AP device driver, its
remove
callback will be invoked for each queue to be removed. The remove
callback
resets the queue and disables IRQ processing. If interrupt
processing was
never enabled for the queue, disabling IRQ processing will fail
resulting
in a kernel OOPS.
This patch verifies IRQ processing is enabled before attempting to
disable
interrupts for the queue.
Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
Signed-off-by: aekrowia <akrowiak@xxxxxxxxxxxxx>
---
 drivers/s390/crypto/vfio_ap_drv.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/s390/crypto/vfio_ap_drv.c
b/drivers/s390/crypto/vfio_ap_drv.c
index be2520cc010b..42d8308fd3a1 100644
--- a/drivers/s390/crypto/vfio_ap_drv.c
+++ b/drivers/s390/crypto/vfio_ap_drv.c
@@ -79,7 +79,8 @@ static void vfio_ap_queue_dev_remove(struct
ap_device *apdev)
ÂÂÂÂÂ apid = AP_QID_CARD(q->apqn);
ÂÂÂÂÂ apqi = AP_QID_QUEUE(q->apqn);
ÂÂÂÂÂ vfio_ap_mdev_reset_queue(apid, apqi, 1);
-ÂÂÂ vfio_ap_irq_disable(q);
+ÂÂÂ if (q->saved_isc != VFIO_AP_ISC_INVALID)
+ÂÂÂÂÂÂÂ vfio_ap_irq_disable(q);
ÂÂÂÂÂ kfree(q);
ÂÂÂÂÂ mutex_unlock(&matrix_dev->lock);
 }
Reset of an APQN does also clear IRQ processing. I don't say that the
resources associated with IRQ handling for the APQN are also cleared.
But when you call PQAP(AQIC) after an PQAP(RAPQ) or PQAP(ZAPQ)
it is superfluous. However, there should not appear any kernel OOPS.
So can you please give me more details about this kernel oops - maybe
I need to add exception handler code to the inline ap_aqic() function.
regards, Harald Freudenberger
Hi Tony,
wasn't it already solved by the patch 5c4c2126Â from Christian ?
No, that patch merely sets the 'matrix_mdev' field of the
'struct vfio_ap_queue' to NULL in the vfio_ap_free_aqic_resources()
function. Also, with the latest master branch which has 5c4c2126
installed, the failure occurs.
Can you send the trace to me please?
[Â 266.989476] crw_info : CRW reports slct=0, oflw=0, chn=0, rsc=B,
anc=0, erc=0, rsid=0
[Â 266.989617] ------------[ cut here ]------------
[Â 266.989622] vfio_ap_wait_for_irqclear: tapq rc 03: 0504
[Â 266.989681] WARNING: CPU: 0 PID: 7 at
drivers/s390/crypto/vfio_ap_ops.c:101 vfio_ap_irq_disable+0x13c/0x1b0
[vfio_ap]
Hi Tony,
This is not a oops this is the warning written in
vfio_ap_wait_for_irqclear() because the AP has been deconfigured.
Note that, IIUC, this (the warning) does not happen for devices bound to
the vfio_ap driver but not currently assigned to a mediated device.
I do not think we should avoid sending a warning in this case because
this is not a normal administration good practice to forcefully take an
AP away like this without smoothly removing the device from the mediated
device.
Regards,
Pierre
[ 266.989682] Modules linked in: xt_CHECKSUM xt_MASQUERADE tun bridge
stp llc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack
ebtable_nat ip6table_nat ip6table_mangle ip6table_raw
ip6table_security iptable_nat nf_nat iptable_mangle iptable_raw
iptable_security nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c
ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables
sunrpc ghash_s390 prng aes_s390 des_s390 libdes vfio_ccw sha512_s390
sha1_s390 eadm_sch zcrypt_cex4 qeth_l2 crc32_vx_s390 dasd_eckd_mod
sha256_s390 qeth sha_common dasd_mod ccwgroup qdio pkey zcrypt vfio_ap
kvm
[Â 266.989704] CPU: 0 PID: 7 Comm: kworker/0:1 Not tainted 5.4.0-rc5 #81
[Â 266.989705] Hardware name: IBM 2964 NE1 749 (LPAR)
[Â 266.989710] Workqueue: events_long ap_scan_bus
[Â 266.989711] Krnl PSW : 0704c00180000000 000003ff8007d89c
(vfio_ap_irq_disable+0x13c/0x1b0 [vfio_ap])
[Â 266.989714]ÂÂÂÂÂÂÂÂÂÂÂ R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3
CC:0 PM:0 RI:0 EA:3
[Â 266.989716] Krnl GPRS: 000000000000000a 0000000000000006
000000000000002b 0000000000000007
[Â 266.989717]ÂÂÂÂÂÂÂÂÂÂÂ 0000000000000007 000000007fe06000
000003ff00000005 0000000000000000
[Â 266.989718]ÂÂÂÂÂÂÂÂÂÂÂ 0000000100000504 0000000000000003
00000001f9d27e40 000003e00003bb5c
[Â 266.989719]ÂÂÂÂÂÂÂÂÂÂÂ 00000001fe765d00 0000000000000504
000003ff8007d898 000003e00003ba60
[Â 266.989724] Krnl Code: 000003ff8007d88c: c02000000ce6ÂÂÂ larl
%r2,3ff8007f258
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 000003ff8007d892: c0e5fffff4c7ÂÂÂ brasl
%r14,3ff8007c220
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ #000003ff8007d898: a7f40001ÂÂÂÂÂÂÂ brc
15,3ff8007d89a
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ >000003ff8007d89c: a7f4ff9dÂÂÂÂÂÂÂ brc
15,3ff8007d7d6
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 000003ff8007d8a0: a7100100 tmlhÂÂÂ %r1,256
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 000003ff8007d8a4: a784ff99ÂÂÂÂÂÂÂ brc
8,3ff8007d7d6
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 000003ff8007d8a8: a7290014 lghiÂÂÂ %r2,20
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 000003ff8007d8ac: c0e5fffff4b0ÂÂÂ brasl
%r14,3ff8007c20c
[Â 266.989772] Call Trace:
[Â 266.989777] ([<000003ff8007d898>] vfio_ap_irq_disable+0x138/0x1b0
[vfio_ap])
[Â 266.989779]Â [<000003ff8007c4d2>]
vfio_ap_queue_dev_remove+0x6a/0x90 [vfio_ap]
[Â 266.989782]Â [<00000000bf0f24f0>] ap_device_remove+0x50/0x110
[Â 266.989784]Â [<00000000beffbaac>]
device_release_driver_internal+0x114/0x1f0
[Â 266.989787]Â [<00000000beff9c88>] bus_remove_device+0x108/0x190
[Â 266.989789]Â [<00000000beff5418>] device_del+0x178/0x3a0
[Â 266.989790]Â [<00000000beff5670>] device_unregister+0x30/0x90
[Â 266.989791]Â [<00000000bf0f0f04>]
__ap_queue_devices_with_id_unregister+0x44/0x50
[Â 266.989793]Â [<00000000beff86ea>] bus_for_each_dev+0x82/0xb0
[Â 266.989794]Â [<00000000bf0f2aba>] ap_scan_bus+0x262/0x878
[Â 266.989798]Â [<00000000beb4785c>] process_one_work+0x1e4/0x410
[Â 266.989800]Â [<00000000beb47ca8>] worker_thread+0x220/0x460
[Â 266.989802]Â [<00000000beb4e99a>] kthread+0x12a/0x160
[Â 266.989805]Â [<00000000bf2d8eb0>] ret_from_fork+0x28/0x2c
[Â 266.989806]Â [<00000000bf2d8eb4>] kernel_thread_starter+0x0/0xc
[Â 266.989807] Last Breaking-Event-Address:
[Â 266.989809]Â [<000003ff8007d898>] vfio_ap_irq_disable+0x138/0x1b0
[vfio_ap]
[Â 266.989810] ---[ end trace 59b4020890dbd391 ]---
Thanks,
Pierre
--
Pierre Morel
IBM Lab Boeblingen