Re: [BUG] WARNING in ib_drain_rq
From: Farhad Alemi
Date: Mon Sep 07 2026 - 22:19:13 EST
Hi Jinpu,
Attached are the reproducer and crash report.
Thanks,
On Thu, Aug 27, 2026 at 11:57 PM Jinpu Wang <jinpu.wang@xxxxxxxxxxxxxxx> wrote:
>
> On Fri, Aug 28, 2026 at 7:52 AM Farhad Alemi <farhad.alemi@xxxxxxxxxxxx> wrote:
> >
> > Hello,
> >
> > As part of the kernel research at ASU's SEFCOM
> > lab, we hit the crash below. Crash reports can be found here:
> >
> > https://github.com/farhad-alemi/public_bug_reports/tree/main/198-warning-in-ib-drain-rq/
> >
> > failed to drain recv queue: -12
> > WARNING: drivers/infiniband/core/verbs.c:2948 at __ib_drain_rq
> > drivers/infiniband/core/verbs.c:2948 [inline], CPU#1: kworker/1:3/5056
> > WARNING: drivers/infiniband/core/verbs.c:2948 at
> > ib_drain_rq+0x30c/0x590 drivers/infiniband/core/verbs.c:3078, CPU#1:
> > kworker/1:3/5056
> > RIP: 0010:ib_drain_rq+0x30f/0x590 drivers/infiniband/core/verbs.c:3078
> > Call Trace:
> > ib_drain_rq+0x30f/0x590 drivers/infiniband/core/verbs.c:3078
> > ib_drain_qp+0x1bb/0x430 drivers/infiniband/core/verbs.c:3102
> > stop_cm drivers/infiniband/ulp/rtrs/rtrs-clt.c:1763 [inline]
> > rtrs_clt_stop_and_destroy_conns+0x216/0x1650
> > drivers/infiniband/ulp/rtrs/rtrs-clt.c:2210
> > rtrs_clt_err_recovery_work+0x67/0xc0
> > drivers/infiniband/ulp/rtrs/rtrs-clt.c:1524
> > process_scheduled_works+0x881/0x1170 kernel/workqueue.c:3397
> > worker_thread+0x988/0xde0 kernel/workqueue.c:3478
> > kthread+0x333/0x420 kernel/kthread.c:436
> > Kernel panic - not syncing: kernel: panic_on_warn set ...
> >
> > Our reproducer.c is available upon request.
> Hi Farhad,
>
> Thanks for the report. Please share the reproducer.c so we can take a
> closer look.
>
> Thanks,
> >
> > Happy to test a patch if that would help.
> >
> > Regards,
Linux version 7.3.0-rc2-00006-g28924df2a08f
[ 59.908453][ T5048] WARNING: drivers/infiniband/core/verbs.c:2960 at ib_drain_rq+0x305/0x470, CPU#1: kworker/1:3/5048
[ 60.013069][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013121][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013164][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013201][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013237][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013273][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013319][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013356][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013395][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013431][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013467][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013503][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013539][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013575][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013611][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013657][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013692][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013743][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013778][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013820][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013856][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013891][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013927][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013962][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.013996][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014043][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014077][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014112][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014147][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014181][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014216][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014251][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014286][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014320][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014355][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014389][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014424][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014459][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014493][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014527][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014562][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014597][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014632][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014683][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014718][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014776][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014818][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014854][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014889][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014925][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014961][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.014997][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015032][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015089][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015125][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015161][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015197][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015232][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015268][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015303][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015339][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015375][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015410][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.015446][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.021834][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.021920][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022004][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022085][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022138][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022191][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022243][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022307][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022368][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.022419][ C0] rtrs_client L663: <s3>: rtrs_post_recv_empty(): -ENOMEM
[ 60.187033][ T5048] Modules linked in:
[ 60.187450][ T5048] CPU: 1 UID: 0 PID: 5048 Comm: kworker/1:3 Not tainted 7.3.0-rc2-00006-g28924df2a08f #1 PREEMPT(full)
[ 60.188499][ T5048] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.17.0-debian-1.17.0-1ubuntu1 04/01/2014
[ 60.189569][ T5048] Workqueue: rtrs_client_wq rtrs_clt_err_recovery_work
[ 60.190258][ T5048] RIP: 0010:ib_drain_rq+0x308/0x470
[ 60.190768][ T5048] Code: 2e e8 60 3f 4b 17 41 89 c4 c6 43 04 f8 31 ff 89 c6 e8 5c f3 10 f9 45 85 e4 74 16 e8 52 ee 10 f9 48 8d 3d ab d6 ad 07 44 89 e6 <67> 48 0f b9 3a eb 6f 48 8b 44 24 10 4c 8d 60 34 4c 89 e0 48 c1 e8
[ 60.192600][ T5048] RSP: 0018:ffffc9000840f3e0 EFLAGS: 00010293
[ 60.193240][ T5048] RAX: ffffffff88b6a91e RBX: fffff52001081e80 RCX: ffff888109582640
[ 60.194019][ T5048] RDX: 0000000000000000 RSI: 00000000fffffff4 RDI: ffffffff90647fd0
[ 60.194784][ T5048] RBP: ffffc9000840f670 R08: ffffffff9056307f R09: 1ffffffff20ac60f
[ 60.195576][ T5048] R10: dffffc0000000000 R11: fffffbfff20ac610 R12: 00000000fffffff4
[ 60.196362][ T5048] R13: dffffc0000000000 R14: ffff8880287d0000 R15: ffffc9000840f560
[ 60.197139][ T5048] FS: 0000000000000000(0000) GS:ffff8881da58b000(0000) knlGS:0000000000000000
[ 60.198027][ T5048] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 60.198669][ T5048] CR2: 00007fe619780af0 CR3: 0000000024308000 CR4: 0000000000752ef0
[ 60.199462][ T5048] PKRU: 55555554
[ 60.199832][ T5048] Call Trace:
[ 60.200159][ T5048] <TASK>
[ 60.200470][ T5048] ? __pfx_ib_drain_rq+0x10/0x10
[ 60.200977][ T5048] ? __pfx_ib_drain_qp_done+0x10/0x10
[ 60.201508][ T5048] ? __pfx_ib_drain_qp_done+0x10/0x10
[ 60.202053][ T5048] ? _raw_spin_unlock_irqrestore+0x4c/0x80
[ 60.202618][ T5048] ? ib_send_cm_dreq+0x407/0x690
[ 60.203153][ T5048] ib_drain_qp+0x1be/0x430
[ 60.203598][ T5048] ? __pfx_ib_drain_qp+0x10/0x10
[ 60.204112][ T5048] ? lockdep_hardirqs_on+0x7b/0x110
[ 60.204616][ T5048] ? enable_work+0x1fd/0x230
[ 60.205073][ T5048] ? __cancel_work_sync+0xf8/0x110
[ 60.205568][ T5048] rtrs_clt_stop_and_destroy_conns+0x215/0x1520
[ 60.206196][ T5048] ? __lock_acquire+0x74c/0x2db0
[ 60.206694][ T5048] ? look_up_lock_class+0x57/0x110
[ 60.207208][ T5048] ? register_lock_class+0x31/0x2e0
[ 60.207728][ T5048] ? __lock_acquire+0x74c/0x2db0
[ 60.208217][ T5048] ? hrtimer_start_range_ns_common+0x844/0xba0
[ 60.208841][ T5048] ? __pfx_rtrs_clt_stop_and_destroy_conns+0x10/0x10
[ 60.209512][ T5048] ? do_raw_spin_lock+0x12b/0x2f0
[ 60.210029][ T5048] ? lock_acquire+0x115/0x350
[ 60.210491][ T5048] rtrs_clt_err_recovery_work+0x67/0xc0
[ 60.211043][ T5048] ? process_scheduled_works+0x94e/0x1900
[ 60.211611][ T5048] process_scheduled_works+0xc99/0x1900
[ 60.212178][ T5048] ? __pfx_process_scheduled_works+0x10/0x10
[ 60.212824][ T5048] ? assign_work+0x3d5/0x5e0
[ 60.213295][ T5048] worker_thread+0xa53/0xfc0
[ 60.213764][ T5048] kthread+0x38b/0x480
[ 60.214211][ T5048] ? __pfx_worker_thread+0x10/0x10
[ 60.214744][ T5048] ? __pfx_kthread+0x10/0x10
[ 60.215250][ T5048] ret_from_fork+0x514/0xb70
[ 60.215703][ T5048] ? __pfx_ret_from_fork+0x10/0x10
[ 60.216224][ T5048] ? __switch_to+0xc79/0x1410
[ 60.216697][ T5048] ? __pfx_kthread+0x10/0x10
[ 60.217173][ T5048] ret_from_fork_asm+0x1a/0x30
[ 60.217643][ T5048] </TASK>
[ 60.217954][ T5048] Kernel panic - not syncing: kernel: panic_on_warn set ...
[ 60.218665][ T5048] CPU: 1 UID: 0 PID: 5048 Comm: kworker/1:3 Not tainted 7.3.0-rc2-00006-g28924df2a08f #1 PREEMPT(full)
[ 60.219752][ T5048] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.17.0-debian-1.17.0-1ubuntu1 04/01/2014
[ 60.220779][ T5048] Workqueue: rtrs_client_wq rtrs_clt_err_recovery_work
[ 60.221436][ T5048] Call Trace:
[ 60.221759][ T5048] <TASK>
[ 60.222053][ T5048] vpanic+0x56d/0xa60
[ 60.222445][ T5048] ? __pfx__printk+0x10/0x10
[ 60.222920][ T5048] ? __pfx_vpanic+0x10/0x10
[ 60.223370][ T5048] ? is_bpf_text_address+0x292/0x2b0
[ 60.223904][ T5048] ? is_bpf_text_address+0x26/0x2b0
[ 60.224445][ T5048] panic+0xc5/0xd0
[ 60.224824][ T5048] ? __pfx_panic+0x10/0x10
[ 60.225271][ T5048] ? ret_from_fork_asm+0x1a/0x30
[ 60.225771][ T5048] __warn+0x315/0x4c0
[ 60.226185][ T5048] ? ib_drain_rq+0x305/0x470
[ 60.226646][ T5048] ? ib_drain_rq+0x305/0x470
[ 60.227109][ T5048] __report_bug+0x273/0x580
[ 60.227576][ T5048] ? ib_drain_rq+0x305/0x470
[ 60.228027][ T5048] ? __pfx___report_bug+0x10/0x10
[ 60.228517][ T5048] ? do_raw_spin_lock+0x12b/0x2f0
[ 60.229062][ T5048] report_bug_entry+0x19b/0x290
[ 60.229561][ T5048] ? ib_drain_rq+0x308/0x470
[ 60.230031][ T5048] ? ib_drain_rq+0x30d/0x470
[ 60.230477][ T5048] handle_bug+0xce/0x200
[ 60.230892][ T5048] exc_invalid_op+0x1a/0x50
[ 60.231332][ T5048] asm_exc_invalid_op+0x1a/0x20
[ 60.231805][ T5048] RIP: 0010:ib_drain_rq+0x308/0x470
[ 60.232308][ T5048] Code: 2e e8 60 3f 4b 17 41 89 c4 c6 43 04 f8 31 ff 89 c6 e8 5c f3 10 f9 45 85 e4 74 16 e8 52 ee 10 f9 48 8d 3d ab d6 ad 07 44 89 e6 <67> 48 0f b9 3a eb 6f 48 8b 44 24 10 4c 8d 60 34 4c 89 e0 48 c1 e8
[ 60.234188][ T5048] RSP: 0018:ffffc9000840f3e0 EFLAGS: 00010293
[ 60.234769][ T5048] RAX: ffffffff88b6a91e RBX: fffff52001081e80 RCX: ffff888109582640
[ 60.235529][ T5048] RDX: 0000000000000000 RSI: 00000000fffffff4 RDI: ffffffff90647fd0
[ 60.236310][ T5048] RBP: ffffc9000840f670 R08: ffffffff9056307f R09: 1ffffffff20ac60f
[ 60.237085][ T5048] R10: dffffc0000000000 R11: fffffbfff20ac610 R12: 00000000fffffff4
[ 60.237881][ T5048] R13: dffffc0000000000 R14: ffff8880287d0000 R15: ffffc9000840f560
[ 60.238657][ T5048] ? ib_drain_rq+0x2fe/0x470
[ 60.239148][ T5048] ? __pfx_ib_drain_rq+0x10/0x10
[ 60.239637][ T5048] ? __pfx_ib_drain_qp_done+0x10/0x10
[ 60.240158][ T5048] ? __pfx_ib_drain_qp_done+0x10/0x10
[ 60.240679][ T5048] ? _raw_spin_unlock_irqrestore+0x4c/0x80
[ 60.241248][ T5048] ? ib_send_cm_dreq+0x407/0x690
[ 60.241730][ T5048] ib_drain_qp+0x1be/0x430
[ 60.242168][ T5048] ? __pfx_ib_drain_qp+0x10/0x10
[ 60.242654][ T5048] ? lockdep_hardirqs_on+0x7b/0x110
[ 60.243191][ T5048] ? enable_work+0x1fd/0x230
[ 60.243653][ T5048] ? __cancel_work_sync+0xf8/0x110
[ 60.244175][ T5048] rtrs_clt_stop_and_destroy_conns+0x215/0x1520
[ 60.244792][ T5048] ? __lock_acquire+0x74c/0x2db0
[ 60.245281][ T5048] ? look_up_lock_class+0x57/0x110
[ 60.245776][ T5048] ? register_lock_class+0x31/0x2e0
[ 60.246311][ T5048] ? __lock_acquire+0x74c/0x2db0
[ 60.246809][ T5048] ? hrtimer_start_range_ns_common+0x844/0xba0
[ 60.247424][ T5048] ? __pfx_rtrs_clt_stop_and_destroy_conns+0x10/0x10
[ 60.248066][ T5048] ? do_raw_spin_lock+0x12b/0x2f0
[ 60.248555][ T5048] ? lock_acquire+0x115/0x350
[ 60.249043][ T5048] rtrs_clt_err_recovery_work+0x67/0xc0
[ 60.249579][ T5048] ? process_scheduled_works+0x94e/0x1900
[ 60.250134][ T5048] process_scheduled_works+0xc99/0x1900
[ 60.250682][ T5048] ? __pfx_process_scheduled_works+0x10/0x10
[ 60.251287][ T5048] ? assign_work+0x3d5/0x5e0
[ 60.251750][ T5048] worker_thread+0xa53/0xfc0
[ 60.252223][ T5048] kthread+0x38b/0x480
[ 60.252638][ T5048] ? __pfx_worker_thread+0x10/0x10
[ 60.253145][ T5048] ? __pfx_kthread+0x10/0x10
[ 60.253623][ T5048] ret_from_fork+0x514/0xb70
[ 60.254099][ T5048] ? __pfx_ret_from_fork+0x10/0x10
[ 60.254594][ T5048] ? __switch_to+0xc79/0x1410
[ 60.255058][ T5048] ? __pfx_kthread+0x10/0x10
[ 60.255511][ T5048] ret_from_fork_asm+0x1a/0x30
[ 60.255984][ T5048] </TASK>
[ 60.256571][ T5048] Kernel Offset: disabled
[ 60.257131][ T5048] Rebooting in 86400 seconds..
Attachment:
reproducer.c
Description: Binary data