RE: [PATCH v5 22/27] vfio/cxl: Clear the HDM access gate after a hot reset

[Manish Honap]

> -----Original Message-----
> From: Alex Williamson <alex@xxxxxxxxxxx>
> Sent: Tuesday, September 22, 2026 7:43 AM
> To: Manish Honap <mhonap@xxxxxxxxxx>
> Cc: jgg@xxxxxxxx; Ankit Agrawal <ankita@xxxxxxxxxx>; jic23@xxxxxxxxxx;
> dave.jiang@xxxxxxxxx; alejandro.lucero-palau@xxxxxxx; Srirangan
> Madhavan <smadhavan@xxxxxxxxxx>; corbet@xxxxxxx;
> skhan@xxxxxxxxxxxxxxxxxxx; dave@xxxxxxxxxxxx; alison.schofield@xxxxxxxxx;
> vishal.l.verma@xxxxxxxxx; iweiny@xxxxxxxxxx; ming.li@xxxxxxxxxxxx; Yishai
> Hadas <yishaih@xxxxxxxxxx>; Shameer Kolothum Thodi
> <skolothumtho@xxxxxxxxxx>; kevin.tian@xxxxxxxxx; bhelgaas@xxxxxxxxxx;
> dmatlack@xxxxxxxxxx; kees@xxxxxxxxxx; gustavoars@xxxxxxxxxx; Neo Jia
> <cjia@xxxxxxxxxx>; Krishnakant Jaju <kjaju@xxxxxxxxxx>; Vikram Sethi
> <vsethi@xxxxxxxxxx>; Zhi Wang <zhiw@xxxxxxxxxx>; linux-
> doc@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx;
> linux-cxl@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-
> kselftest@xxxxxxxxxxxxxxx; linux-hardening@xxxxxxxxxxxxxxx; alex@xxxxxxxxxxx
> Subject: Re: [PATCH v5 22/27] vfio/cxl: Clear the HDM access gate after a hot
> reset
> 
> External email: Use caution opening links or attachments
> 
> 
> On Thu, 17 Sep 2026 00:05:35 +0530
> <mhonap@xxxxxxxxxx> wrote:
> 
> > From: Manish Honap <mhonap@xxxxxxxxxx>
> >
> > The HDM memory fault inserts a PFN only when the decoder is in a
> > known-good state (cxl->hdm_valid), so the host CPU never faults the
> > coherent range into a disabled decoder.
> >
> > VFIO_DEVICE_PCI_HOT_RESET drives a plain secondary bus reset rather
> > than the CXL reset sequence, so nothing restores the decoder in the
> > reset_done handler. A secondary bus reset only reaches a CXL endpoint
> > when the upstream port has SBR unmasked (CXL r3.1 sec 8.1.5.2); in
> > that case the reset decommits the decoder, so clear cxl->hdm_valid to
> > keep a later fault from inserting a PFN into a dead decoder.
> >
> > A subsequent VFIO_DEVICE_RESET runs the CXL reset sequence and
> > restores it, and a masked SBR is a no-op that leaves the decoder intact.
> >
> > vfio_cxl_sbr_unmasked() mirrors the cxl_sbr_masked() check the PCI
> > core uses (drivers/pci/pci.c) for its own CXL bus reset.
> 
> This all sounds very broken.  If a hot reset is masked, it should fail, not silently
> do nothing.  If a hot reset is unmasked, we can't just invent that the user
> needs to follow it with a reset ioctl, and the decoder is dead for the remainder
> of their session otherwise.
> 
> So on one hand, we have a reset we can't use because it doesn't do anything,
> and on the other we have a reset that we can't use because in doing
> something it makes the device effectively unrecoverable, without some
> assumption to perform a reset ioctl.
> 
> Are bus resets just not supported on CXL devices through vfio-pci?
> Thanks,

I will try to handle both these cases in v6 correctly:

1. Masked
with Unmask SBR clear in the port's CXL DVSEC, the SBR does nothing to the
device (CXL r3.1 sec 8.1.5.2). The PCI core already refuses pci_reset_bus_function()
in that case. I will add the same check to the bus and slot reset paths and their probes, so
VFIO_DEVICE_GET_PCI_HOT_RESET_INFO and VFIO_DEVICE_PCI_HOT_RESET fail for the
device instead of doing nothing. A VMM will then use VFIO_DEVICE_RESET, which runs the
CXL reset. I will add the PCI core change in its own patch.

2. Unmasked
The reset clears the decoder, so vfio-cxl's reset_done restores it with the same cxl-core
helper the PCI core's cxl_bus reset method calls after its SBR, and reopens the HDM gate. If the
restore fails, the hot reset ioctl returns the error. No follow-up  VFIO_DEVICE_RESET is needed.

> 
> Alex
> 
> 
> > Assisted-by: LLM
> > Signed-off-by: Manish Honap <mhonap@xxxxxxxxxx>
> > ---
> >  drivers/vfio/pci/cxl/vfio_cxl_core.c | 38
> > ++++++++++++++++++++++++++++
> >  1 file changed, 38 insertions(+)
> >
> > diff --git a/drivers/vfio/pci/cxl/vfio_cxl_core.c
> > b/drivers/vfio/pci/cxl/vfio_cxl_core.c
> > index 37e8a3b54cfb..395b4a5b0956 100644
> > --- a/drivers/vfio/pci/cxl/vfio_cxl_core.c
> > +++ b/drivers/vfio/pci/cxl/vfio_cxl_core.c
> > @@ -514,8 +514,46 @@ static void vfio_cxl_reset_prepare(struct
> > vfio_pci_core_device *vdev)  {  }
> >
> > +/*
> > + * A secondary bus reset only reaches a CXL endpoint when the
> > +upstream port
> > + * has SBR unmasked (CXL r3.1 sec 8.1.5.2); otherwise the Bridge
> > +Control SBR
> > + * bit is ignored and the decoder is left intact. Mirrors the
> > +cxl_sbr_masked()
> > + * check the PCI core uses for its own CXL bus reset.
> > + */
> > +static bool vfio_cxl_sbr_unmasked(struct pci_dev *pdev) {
> > +     struct pci_dev *bridge = pci_upstream_bridge(pdev);
> > +     u16 dvsec, ctl;
> > +
> > +     if (!bridge)
> > +             return false;
> > +
> > +     dvsec = pci_find_dvsec_capability(bridge, PCI_VENDOR_ID_CXL,
> > +                                       PCI_DVSEC_CXL_PORT);
> > +     if (!dvsec)
> > +             return false;
> > +
> > +     if (pci_read_config_word(bridge, dvsec + PCI_DVSEC_CXL_PORT_CTL,
> &ctl))
> > +             return false;
> > +
> > +     return ctl & PCI_DVSEC_CXL_PORT_CTL_UNMASK_SBR;
> > +}
> > +
> >  static void vfio_cxl_reset_done(struct vfio_pci_core_device *vdev)  {
> > +     struct vfio_cxl_state *cxl = vdev->cxl;
> > +
> > +     /*
> > +      * VFIO_DEVICE_PCI_HOT_RESET drives a plain secondary bus reset, not
> the
> > +      * CXL-aware cxl_reset_bus_function(), so nothing restores the HDM
> > +      * decoder here. When the upstream port has SBR unmasked the reset
> > +      * decommits the decoder; gate host access to the HDM range so a later
> > +      * fault cannot insert a PFN into a dead decoder. A VFIO_DEVICE_RESET
> > +      * then runs the CXL reset sequence and restores it. A masked SBR is a
> > +      * no-op and leaves the decoder intact.
> > +      */
> > +     if (vfio_cxl_sbr_unmasked(vdev->pdev))
> > +             cxl->hdm_valid = false;
> >  }
> >
> >  static const struct vfio_cxl_ops vfio_cxl_ops = {