Re: [PATCH 0/5] PCI/CXL: Save and restore CXL DVSEC and HDM state across resets

[Dan Williams]

Alex Williamson wrote:

Hey Alex, sorry for the lag in responding here...

> On Tue, 17 Mar 2026 10:03:28 -0700
> Dan Williams <dan.j.williams@xxxxxxxxx> wrote:
> 
> > Manish Honap wrote:
> > [..]
> > > > The CXL accelerator series is currently contending with being able to
> > > > restore device configuration after reset. I expect vfio-cxl to build on
> > > > that, not push CXL flows into the PCI core.  
> > > 
> > > Hello Dan,
> > > 
> > > My VFIO CXL Type-2 passthrough series [1] takes a position on this that I
> > > would like to explain because I expect you will have similar concerns about
> > > it and I'd rather have this conversation now.
> > > 
> > > Type-2 passthrough series takes the opposite structural approach as you are
> > > suggesting here: CXL Type-2 support is an optional extension compiled into
> > > vfio-pci-core (CONFIG_VFIO_CXL_CORE), not a separate driver.
> > > 
> > > Here is the reasoning:
> > > 
> > > 1. Device enumeration
> > > =====================
> > > 
> > > CXL Type-2 devices (GPU + accelerator class) are enumerated as struct pci_dev
> > > objects.  The kernel discovers them through PCI config space scan, not through
> > > the CXL bus. The CXL capability is advertised via the DVSEC (PCI_EXT_CAP_ID
> > > 0x23, Vendor ID 0x1E98), which is PCI config space. There is no CXL bus
> > > device to bind to.
> > > 
> > > A standalone vfio-cxl driver would therefore need to match on the PCI device
> > > just like vfio-pci does, and then call into vfio-pci-core for every PCI
> > > concern: config space emulation, BAR region handling, MSI/MSI-X, INTx, DMA
> > > mapping, FLR, and migration callbacks. That is the variant driver pattern
> > > we rejected in favour of generic CXL passthrough. We have seen this exact  
> > 
> > Lore link for this "rejection" discussion?
> > 
> > > outcome with the prior iterations of this series before we moved to the
> > > enlightened vfio-pci model.  
> > 
> > I still do not understand the argument. CXL functionality is a library
> > that PCI drivers can use.
> 
[..]
> If we were to make "vfio-cxl" as a vfio-pci variant driver, we'd need
> to expand the ID table for specific devices, which becomes a
> maintenance issue.  Otherwise userspace would need to detect the CXL
> capabilities and override the automatic driver aliases.  We can't match
> drivers based on DVSEC capabilities and we don't have any protocol to
> define a "2nd best" match for a device alias if probe fails.

I can see the argument, and why it makes sense to attempt this way
first. Point conceded.

Now a follow on concern is the plan to manage a case of "PCI operation
is available, but CXL operation is not. Does the driver proceed?" Put
another way, I immediately see how to convey the policy of "continue
without CXL" when there is an explicit driver distinction, but it is
ambiguous with an enlightened vfio-pci driver.

> > If vfio-pci functionality is also a library
> > then vfio-cxl is a driver that uses services from both libraries. Where
> > the module and driver name boundaries are drawn is more an organization
> > decision not an functional one.
> 
> But as above, it is functional.  Someone needs to define when to use
> which driver, which leads to libvirt needing to specify whether a
> device is being exposed as PCI or CXL, and the same understanding in
> each VMM.  OTOH, using vfio-pci as the basis and layering CXL feature
> detection, ie. enlightenment, gives us a more compatible, incremental
> approach.

Ok, to make sure I understand the proposal: userspace still needs to to
end up with knowledge of CXL operation, but that need not be resolved by
module policy.

Userspace also just needs to be ok with the unsightliness of the CXL
modules autoloading on systems without CXL.

> > The argument for vfio-cxl organizational independence is more about
> > being able to tell at a diffstat level the relative PCI vs CXL
> > maintenance impact / regression risk.
> 
> But we still have that.  CXL enlightenment for vfio-pci(-core) can
> still be configured out and compartmentalized into separate helper
> library code.

Yes, modulo some of the proposal here to enlighten the PCI core with CXL
specifics that I want to give more scrutiny.

> > > 2. CXL-CORE involvement
> > > =======================
> > > 
> > > CXL type-2 passthrough series does not bypass CXL core. At vfio_pci_probe()
> > > time the CXL enlightenment layer:
> > > 
> > >   - calls cxl_get_hdm_info() to probe the HDM Decoder Capability block,
> > >   - calls cxl_get_committed_decoder() to locate pre-committed firmware regions,
> > >   - calls cxl_create_region() / cxl_request_dpa() for dynamic allocation,
> > >   - creates a struct cxl_memdev via the CXL core (via cxl_probe_component_regs,
> > >     the same path Alejandro's v23 series uses).
> > > 
> > > The CXL core is fully involved.  The difference is that the binding to
> > > userspace is still through vfio-pci, which already manages the pci_dev
> > > lifecycle, reset sequencing, and VFIO region/irq API.  
> > 
> > Sure, every CXL driver in the system will do the same.
> > 
> > > 3. Standalone vfio-cxl
> > > ======================
> > > 
> > > To match the model you are suggesting, vfio-cxl would need to:
> > > 
> > >   (a) Register a new driver on the CXL bus (struct cxl_driver), probing
> > >       struct cxl_memdev or a new struct cxl_endpoint,  
> > 
> > What, why? Just like this patch was series was proposing extending the
> > PCI core with additional common functionality the proposal is extend the
> > CXL core object drivers with the same.
> 
> I don't follow, what is the proposal?

Implement features like CXL Reset as operations against CXL objects like
memdevs and regions. For example, PCI reset does not consider management
of cache coherent memory, and certainly not interleaved cache coherent
memory. Other CXL drivers also benefit if these capabilities are
centralized.

> > >   (b) Re-implement or delegate everything vfio-pci-core provides — config
> > >       space, BAR regions, IRQs, DMA, FLR, and VFIO container management —
> > 
> > What is the argument against a library?
> 
> vfio-pci-core is already a library, the extensions to support CXL as an
> enlightenment of vfio-pci is also a library.  The issue is that a
> vfio-cxl PCI driver module presents more issues than simply code
> organization.

Understood. As I conceded above my concerns are complications that a
vfio-cxl module does not solve cleanly.

> > >   (c) present to userspace through a new device model distinct from
> > >       vfio-pci.  
> > 
> > CXL is a distinct operational model. What breaks if userspace is
> > required to explicitly account for CXL passhthrough?
> 
> The entire virtualization stack needs to gain an understanding of the
> intended use case of the device rather than simply push a PCI device
> with CXL capabilities out to the guest.

Agree.

> > > This is a significant new surface. QEMU's CXL passthrough support already
> > > builds on vfio-pci: it receives the PCI device via VFIO, reads the
> > > VFIO_DEVICE_INFO_CAP_CXL capability chain, and exposes the CXL topology.
> > > A vfio-cxl object model would require non-trivial QEMU changes for something
> > > that already works in the enlightened vfio-pci model.  
> > 
> > What specifically about a kernel code organization choice affects the
> > QEMU implementation? A uAPI is kernel code organization agnostic.
> > 
> > The concern is designing ourselves into a PCI corner when longterm QEMU
> > benefits from understanding CXL objects. For example, CXL error handling
> > / recovery is already well on its way to being performed in terms of CXL
> > port objects.
> 
> Are you suggesting that rather than using the PCI device as the basis
> for assignment to a userspace driver or VM that we make each port
> objects assignable and somehow collect them into configuration on top of
> a PCI device?  I don't think these port objects are isolated for such a
> use case.  I'd like to better understand how you envision this to work.

No, simply that CXL operations relative to that assigned PCI device are
serviced by the CXL core. The object to manage over reset is subject to
CPU speculative reads and potentially interleave, I think it breaks the
PCI expectations of local device scope operations.

If CXL Reset in particular stays out of the PCI core it at least
requires something CXL enlightened to be loaded, and at a minimum I do
not think that "something CXL enlightened" should be the PCI core.

There is a reason the CXL specification decided to block secondary bus
reset by default.

> The organization of the code in the kernel seems 90%+ the same whether
> we enlighten vfio-pci to detect and expose CXL features or we create a
> separate vfio-cxl PCI driver only for CXL devices, but the userspace
> consequences are increased significantly.

Agree.

> > > 4. Module dependency
> > > ====================
> > > 
> > > Current solution: CONFIG_VFIO_CXL_CORE depends on CONFIG_CXL_BUS. We do not
> > > add CXL knowledge to the PCI core;  
> > 
> > drivers/pci/cxl.c
> 
> This is largely a consequence of CXL_BUS being a loadable module.

Yes, the question is why does that matter for CXL enlightened operation?
Simply do not burden the PCI core to learn all the CXL concerns.