[PATCH iwl-net v2 0/5] ice: fix RTNL vs auxiliary device lock inversion

[Petr Oros]

ice takes the auxiliary device lock of its RDMA auxiliary device from
three places that run with the RTNL lock held. The RDMA auxiliary probe
takes the two in the opposite order, because __driver_attach() holds the
auxiliary device lock across ib_register_device(), which needs the RTNL
lock. The result is a hard deadlock of everything that needs the RTNL
lock, so the boot never completes:

  systemd-udevd   state:D
    really_probe                  <- holds device_lock(&adev->dev)
    icrdma_probe
    irdma_ib_register_device
    ib_register_device
    setup_device
    irdma_roce_port_immutable
    ib_query_port
    irdma_query_port
    ib_get_eth_speed              <- blocks on rtnl_lock()

  lldpad          state:D
    rtnetlink_rcv_msg             <- holds rtnl_lock()
    dcb_doit
    dcbnl_ieee_set
    ice_dcbnl_setets
    ice_pf_dcb_cfg
    ice_send_event_to_aux         <- blocks on device_lock(&adev->dev)

Lockdep cannot flag it, because device_initialize() marks dev->mutex
with lockdep_set_novalidate_class(). A PREEMPT_RT kernel with
CONFIG_DEBUG_RT_MUTEXES reports "rtmutex deadlock detected" instead.

The probe side cannot be changed, so the three ice callers use
device_trylock() and refuse the operation with -EBUSY when the driver
core holds the lock. Patches 1 and 2 are new in v2 and fix two separate
problems on the same sites, the auxiliary device pointer was tested
before pf->adev_mutex was taken, and ndo_setup_tc() took pf->adev_mutex
before pf->tc_mutex while the dcbnl handlers take them the other way
round. Patch 3 keeps the locks across the whole DCB reconfiguration so
that BEFORE_TC_CHANGE and AFTER_TC_CHANGE are either both delivered or
the request is refused before anything changes, and re-arms the service
task events. Patches 4 and 5 turn the remaining two locks into trylocks.

The proper fix is provider-owned registration as in i40e and mlx5, which
makes the inversion impossible by construction. That is a net-next change
across ice, irdma and iidc_rdma.h, this series is the backportable part.

Tested on an E810-XXV with two PFs, irdma bound to both auxiliary
devices, on net tip commit 191aa41548b2d2 ("MAINTAINERS: add Bobby
Eshleman as vsock reviewer") with and without this series. A reproducer
binds the irdma auxiliary driver while driving one of the three RTNL
held callers into the probe window:

                        without the series   with the series
  ethtool set_channels  deadlock 10/10       0/10, window entered 300x
  ndo_setup_tc mqprio   deadlock             0/10, window entered 50x
  dcbnl ets set         deadlock             0/10, window entered 30x

A window entry is a contended device_trylock(), so each one used to
close the cycle. 25 reboots on the patched kernel were clean.

Changes since v1:
- new patch 1 for the auxiliary device pointer race
- new patch 2 for the pf->tc_mutex order in ndo_setup_tc()
- patch 3 holds the auxiliary device locks across the whole DCB
  reconfiguration and sends AFTER_TC_CHANGE under them instead of
  dropping it when the lock is contended, and takes the locks before it
  consumes pf->oicr_err_reg in the service task
- patch 5 warns instead of netdev_dbg(), the qdisc teardown ignores the
  return value

Link to v1:
https://lore.kernel.org/all/20261019065301.3983537-1-poros@xxxxxxxxxx/

Petr Oros (5):
  ice: check the auxiliary device pointer under adev_mutex
  ice: take tc_mutex before the auxiliary device locks in ndo_setup_tc
  ice: avoid RTNL deadlock when notifying the RDMA auxiliary driver
  ice: do not block on the aux device lock in ethtool set_channels
  ice: do not block on the aux device lock in ndo_setup_tc

 drivers/net/ethernet/intel/ice/ice_dcb_lib.c | 46 ++++++++++++++++---
 drivers/net/ethernet/intel/ice/ice_ethtool.c | 26 +++++++----
 drivers/net/ethernet/intel/ice/ice_idc.c     | 48 ++++++++++++++++----
 drivers/net/ethernet/intel/ice/ice_idc_int.h |  2 +
 drivers/net/ethernet/intel/ice/ice_main.c    | 47 +++++++++++++------
 5 files changed, 130 insertions(+), 39 deletions(-)


base-commit: 07bd391eb9572e74f073c49acef327611234f9bb
-- 
2.56.0