Re: [RFC net v4 0/4] bnxt_en: Make RING FREE more robust
From: anmory
Date: Tue Oct 06 2026 - 12:44:04 EST
Hi Joe,
we appear to be hitting a very similar issue to the one described in
this RFC on two Broadcom BCM57504 systems.
One detail that may be particularly relevant is the ordering in our
reproductions: on both systems the AMD-Vi IO_PAGE_FAULT occurs before
the NETDEV WATCHDOG / TX timeout.
We originally encountered the problem after updating from Debian
6.12.107-1 to 6.12.111-1.
We subsequently reproduced it deliberately on two separate systems.
Test results:
affected-host-1:
BCM57504 [14e4:1751], bnxt_en
BIOS: 1.16.2
NIC FW: 36.11.55.00
FW mgmt: 236.1.153.0
6.12.107-1: stable
6.12.111-1: failure reproduced
affected-host-2:
BCM57504 [14e4:1751], bnxt_en
BIOS: 1.18.2
NIC FW: 36.11.73.00
FW mgmt: 236.1.173.0
6.12.107-1: stable
6.12.111-1: failure reproduced
The failure sequence on affected-host-1 was:
06:49:00 AMD-Vi IO_PAGE_FAULT
06:49:14 NETDEV WATCHDOG: transmit queue 0 timed out
06:49:14 TX timeout detected, starting reset task
06:49:18 HWRM/RING_FREE failures begin
06:49:25 HWRM_RING_ALLOC fails
06:49:25 bnxt_init_nic fails
06:49:25 nic open fails
On affected-host-2:
15:29:09 AMD-Vi IO_PAGE_FAULT
15:29:15 NETDEV WATCHDOG: transmit queue 4 timed out
15:29:15 TX timeout detected, starting reset task
15:29:19 HWRM/RING_FREE failures begin
15:29:37 HWRM_RING_ALLOC fails
15:29:37 bnxt_init_nic fails
15:29:37 nic open fails
So in our reproductions the IOMMU fault precedes the TX watchdog by
approximately 6-14 seconds.
We have also observed the timeout on different TX queues across
different occurrences, so it does not appear to be tied to a specific
queue.
Both interfaces were up and operating at 25 Gbit/s before the failure.
The systems use the IOMMU in translated mode.
After the reset attempt fails, the interface cannot be reopened and a
reboot is required.
The same problem therefore reproduces across:
- two physical systems
- two BIOS revisions
- two NIC firmware revisions
- different TX queues
while reverting to 6.12.107-1 has been stable with the same workload.
I've attached sanitized diagnostic output from both reproductions,
including PCI/device information, firmware versions, IOMMU
configuration and the complete failure event sequence.
We have not bisected the changes between 6.12.107 and 6.12.111 yet.
Given that the IO_PAGE_FAULT precedes the watchdog in our case, I
wonder whether this may provide some information about the initial
failure, in addition to the RING_FREE recovery problem addressed by
your RFC.
Please let me know if there are additional diagnostics that would be
useful, or if you would like us to test the RFC series on one of these
systems.
Thanks,
R.
=== SYSTEM ===
hostname: affected-host-1
bios_version: 1.16.2
bios_date: 01/16/2026
eth1_pci_device: 0000:c4:00.2
=== NIC / DRIVER ===
c4:00.2 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb Ethernet [14e4:1751] (rev 12)
DeviceName: Integrated NIC 1 Port 3-1
Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E BCM57504 4x25G OCP3.0 [14e4:5045]
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en
=== DEVLINK ===
pci/0000:c4:00.2:
driver bnxt_en
serial_number <redacted>
versions:
fixed:
board.id BCM957504
asic.id 1751
asic.rev B2
running:
fw.psid 232.0.2
fw 36.11.55.00
fw.mgmt 236.1.153.0
fw.mgmt.api 1.10.3
stored:
fw.psid 232.0.2
fw 36.11.55.00
fw.mgmt 236.1.153.0
=== CAPTURE BASELINE ===
2026-10-06T06:28:26+00:00
affected-host-1
6.12.111+deb13-amd64
1.16.2
01/16/2026
eth1 UP XX:XX:XX:XX:XX:XX <BROADCAST,MULTICAST,UP,LOWER_UP>
=== RELEVANT BOOT / HARDWARE ===
2026-10-06T06:28:25+00:00 affected-host-1 kernel: Linux version 6.12.111+deb13-amd64 (debian-kernel@xxxxxxxxxxxxxxxx) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.111-1 (2026-09-28)
2026-10-06T06:28:25+00:00 affected-host-1 kernel: DMI: Dell Inc. PowerEdge R7615, BIOS 1.16.2 01/16/2026
2026-10-06T06:28:25+00:00 affected-host-1 kernel: pci 0000:c4:00.2: [14e4:1751] type 00 class 0x020000 PCIe Endpoint
2026-10-06T06:28:25+00:00 affected-host-1 kernel: iommu: Default domain type: Translated
2026-10-06T06:28:25+00:00 affected-host-1 kernel: iommu: DMA domain TLB invalidation policy: lazy mode
2026-10-06T06:28:25+00:00 affected-host-1 kernel: pci 0000:c4:00.2: Adding to iommu group 14
2026-10-06T06:28:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth4: Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem cb010000, node addr XX:XX:XX:XX:XX:XX
2026-10-06T06:28:26+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: NIC Link is Up, 25000 Mbps (NRZ) full duplex, Flow control: ON - receive & transmit
2026-10-06T06:28:26+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: FEC autoneg off encoding: Clause 74 BaseR
=== FAILURE EVENTS ===
2026-10-06T06:49:00+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x000f address=0xfb387000 flags=0x0000]
2026-10-06T06:49:14+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: NETDEV WATCHDOG: CPU: 2: transmit queue 0 timed out 5312 ms
2026-10-06T06:49:14+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: TX timeout detected, starting reset task!
2026-10-06T06:49:18+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T06:49:18+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: hwrm_ring_free type 1 failed. rc:fffffff0 err:2
2026-10-06T06:49:21+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T06:49:21+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: hwrm_ring_free type 2 failed. rc:fffffff0 err:0
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: hwrm_ring_free type 4 failed. rc:fffffff0 err:0
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: hwrm_ring_alloc type 1 failed. rc:ffffffe4 err:4
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: bnxt_init_nic err: fffffffb
2026-10-06T06:49:25+00:00 affected-host-1 kernel: bnxt_en 0000:c4:00.2 eth1: nic open fail (rc: fffffffb)
=== SYSTEM ===
hostname: affected-host-2
bios_version: 1.18.2
bios_date: 07/22/2026
eth1_pci_device: 0000:c4:00.3
=== NIC / DRIVER ===
c4:00.3 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57504 NetXtreme-E 10Gb/25Gb/40Gb/50Gb/100Gb Ethernet [14e4:1751] (rev 12)
DeviceName: Integrated NIC 1 Port 4-1
Subsystem: Broadcom Inc. and subsidiaries NetXtreme-E BCM57504 4x25G OCP3.0 [14e4:5045]
Kernel driver in use: bnxt_en
Kernel modules: bnxt_en
=== DEVLINK ===
pci/0000:c4:00.3:
driver bnxt_en
serial_number <redacted>
versions:
fixed:
board.id BCM957504
asic.id 1751
asic.rev B2
running:
fw.psid 232.0.2
fw 36.11.73.00
fw.mgmt 236.1.173.0
fw.mgmt.api 1.10.3
stored:
fw.psid 232.0.2
fw 36.11.73.00
fw.mgmt 236.1.173.0
=== CAPTURE BASELINE ===
2026-10-06T15:29:09+00:00
affected-host-2
6.12.111+deb13-amd64
1.18.2
07/22/2026
eth1 UP XX:XX:XX:XX:XX:XX <BROADCAST,MULTICAST,UP,LOWER_UP>
=== RELEVANT BOOT / HARDWARE ===
2026-10-06T15:29:07+00:00 affected-host-2 kernel: Linux version 6.12.111+deb13-amd64 (debian-kernel@xxxxxxxxxxxxxxxx) (x86_64-linux-gnu-gcc-14 (Debian 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 SMP PREEMPT_DYNAMIC Debian 6.12.111-1 (2026-09-28)
2026-10-06T15:29:07+00:00 affected-host-2 kernel: DMI: Dell Inc. PowerEdge R7615, BIOS 1.18.2 07/22/2026
2026-10-06T15:29:07+00:00 affected-host-2 kernel: pci 0000:c4:00.3: [14e4:1751] type 00 class 0x020000 PCIe Endpoint
2026-10-06T15:29:07+00:00 affected-host-2 kernel: iommu: Default domain type: Translated
2026-10-06T15:29:07+00:00 affected-host-2 kernel: iommu: DMA domain TLB invalidation policy: lazy mode
2026-10-06T15:29:07+00:00 affected-host-2 kernel: pci 0000:c4:00.3: Adding to iommu group 15
2026-10-06T15:29:07+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth5: Broadcom BCM57504 NetXtreme-E 10Gb/25Gb/50Gb/100Gb/200Gb Ethernet found at mem cb000000, node addr XX:XX:XX:XX:XX:XX
2026-10-06T15:29:09+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: NIC Link is Up, 25000 Mbps (NRZ) full duplex, Flow control: ON - receive & transmit
2026-10-06T15:29:09+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: FEC autoneg off encoding: Clause 74 BaseR
=== FAILURE EVENTS ===
2026-10-06T15:29:09+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0010 address=0xfb526000 flags=0x0000]
2026-10-06T15:29:15+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: NETDEV WATCHDOG: CPU: 23: transmit queue 4 timed out 5628 ms
2026-10-06T15:29:15+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: TX timeout detected, starting reset task!
2026-10-06T15:29:19+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T15:29:19+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 1 failed. rc:fffffff0 err:0
2026-10-06T15:29:19+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 1 failed. rc:ffffffea err:2
2026-10-06T15:29:19+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 1 failed. rc:ffffffea err:2
2026-10-06T15:29:22+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T15:29:22+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 1 failed. rc:fffffff0 err:2
2026-10-06T15:29:22+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 1 failed. rc:ffffffea err:2
2026-10-06T15:29:26+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T15:29:26+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 2 failed. rc:fffffff0 err:0
2026-10-06T15:29:30+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T15:29:30+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 4 failed. rc:fffffff0 err:0
2026-10-06T15:29:33+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T15:29:33+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 2 failed. rc:fffffff0 err:0
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: Resp cmpl intr err msg: 0x51
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 4 failed. rc:fffffff0 err:0
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_free type 0 failed. rc:fffffffb err:1
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: hwrm_ring_alloc type 1 failed. rc:ffffffe4 err:4
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: bnxt_init_nic err: fffffffb
2026-10-06T15:29:37+00:00 affected-host-2 kernel: bnxt_en 0000:c4:00.3 eth1: nic open fail (rc: fffffffb)