RE: [Intel-wired-lan] [PATCH iwl-net 4/4] igb: fix igb_msix_other() handling for PREEMPT_RT

[Romanowski, Rafal]

> -----Original Message-----
> From: Intel-wired-lan <intel-wired-lan-bounces@xxxxxxxxxx> On Behalf Of
> Wander Lairson Costa
> Sent: Wednesday, December 4, 2024 12:42 PM
> To: Nguyen, Anthony L <anthony.l.nguyen@xxxxxxxxx>; Kitszel, Przemyslaw
> <przemyslaw.kitszel@xxxxxxxxx>; Andrew Lunn <andrew+netdev@xxxxxxx>;
> David S. Miller <davem@xxxxxxxxxxxxx>; Eric Dumazet
> <edumazet@xxxxxxxxxx>; Jakub Kicinski <kuba@xxxxxxxxxx>; Paolo Abeni
> <pabeni@xxxxxxxxxx>; Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>; Clark
> Williams <clrkwllms@xxxxxxxxxx>; Steven Rostedt <rostedt@xxxxxxxxxxx>; Auke
> Kok <auke-jan.h.kok@xxxxxxxxx>; Jeff Garzik <jgarzik@xxxxxxxxxx>; moderated
> list:INTEL ETHERNET DRIVERS <intel-wired-lan@xxxxxxxxxxxxxxxx>; open
> list:NETWORKING DRIVERS <netdev@xxxxxxxxxxxxxxx>; open list <linux-
> kernel@xxxxxxxxxxxxxxx>; open list:Real-time Linux
> (PREEMPT_RT):Keyword:PREEMPT_RT <linux-rt-devel@xxxxxxxxxxxxxxx>
> Cc: Wander Lairson Costa <wander@xxxxxxxxxx>; Yuying Ma
> <yuma@xxxxxxxxxx>
> Subject: [Intel-wired-lan] [PATCH iwl-net 4/4] igb: fix igb_msix_other() handling
> for PREEMPT_RT
> 
> During testing of SR-IOV, Red Hat QE encountered an issue where the ip link up
> command intermittently fails for the igbvf interfaces when using the PREEMPT_RT
> variant. Investigation revealed that e1000_write_posted_mbx returns an error
> due to the lack of an ACK from e1000_poll_for_ack.
> 
> The underlying issue arises from the fact that IRQs are threaded by default under
> PREEMPT_RT. While the exact hardware details are not available, it appears that
> the IRQ handled by igb_msix_other must be processed before
> e1000_poll_for_ack times out. However, e1000_write_posted_mbx is called with
> preemption disabled, leading to a scenario where the IRQ is serviced only after
> the failure of e1000_write_posted_mbx.
> 
> Commit 338c4d3902fe ("igb: Disable threaded IRQ for igb_msix_other") forced
> the ISR to run in a non-threaded context. However, Sebastian observed that some
> functions called within the ISR acquire locks that may sleep.
> 
> In the previous two patches, we managed to make igb_msg_mask() safe to call
> from an interrupt context.
> 
> In this commit, we move most of the ISR handling to an interrupt context, leaving
> non IRQ safe code to be called from the thread context under PREEMPT_RT.
> 
> Reproducer:
> 
> ipaddr_vlan=3
> nic_test=ens14f0
> vf=${nic_test}v0 # The main testing steps:
> while true; do
>     ip link set ${nic_test} mtu 1500
>     ip link set ${vf} mtu 1500
>     ip link set $vf up
>     # 3. set vlan and ip for VF
>     ip link set ${nic_test} vf 0 vlan ${ipaddr_vlan}
>     ip addr add 172.30.${ipaddr_vlan}.1/24 dev ${vf}
>     ip addr add 2021:db8:${ipaddr_vlan}::1/64 dev ${vf}
>     # 4. check the link state for VF and PF
>     ip link show ${nic_test}
>     if ! ip link show $vf | grep 'state UP'; then
>         echo 'Error found'
>         break
>     fi
>     ip link set $vf down
> done
> 
> You can also reproduce it more reliably by setting nr_cpus=1 in the kernel
> command line.
> 
> Fixes: 9d5c824399de ("igb: PCI-Express 82575 Gigabit Ethernet driver")
> Signed-off-by: Wander Lairson Costa <wander@xxxxxxxxxx>
> Reported-by: Yuying Ma <yuma@xxxxxxxxxx>
> ---
>  drivers/net/ethernet/intel/igb/igb_main.c | 35 ++++++++++++++++-------
>  1 file changed, 24 insertions(+), 11 deletions(-)
> 
> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c
> b/drivers/net/ethernet/intel/igb/igb_main.c
> index 5828831fd29c2..b2894cebe2c9e 100644
> --- a/drivers/net/ethernet/intel/igb/igb_main.c
> +++ b/drivers/net/ethernet/intel/igb/igb_main.c
> @@ -131,6 +131,7 @@ static void igb_set_uta(struct igb_adapter *adapter, bool

Tested-by: Rafal Romanowski <rafal.romanowski@xxxxxxxxx>