RE: [PATCH v6 07/31] vfio/pci: Detect CXL devices and load the CXL provider on demand

From: Manish Honap

Date: Wed Nov 18 2026 - 14:27:03 EST




> -----Original Message-----
> From: Alex Williamson <alex@xxxxxxxxxxx>
> Sent: Thursday, November 5, 2026 5:23 AM
> To: Manish Honap <mhonap@xxxxxxxxxx>
> Cc: jgg@xxxxxxxx; Ankit Agrawal <ankita@xxxxxxxxxx>; jic23@xxxxxxxxxx;
> dave.jiang@xxxxxxxxx; alejandro.lucero-palau@xxxxxxx; Srirangan Madhavan
> <smadhavan@xxxxxxxxxx>; corbet@xxxxxxx; skhan@xxxxxxxxxxxxxxxxxxx;
> dave@xxxxxxxxxxxx; alison.schofield@xxxxxxxxx; vishal.l.verma@xxxxxxxxx;
> iweiny@xxxxxxxxxx; ming.li@xxxxxxxxxxxx; Yishai Hadas
> <yishaih@xxxxxxxxxx>; Shameer Kolothum Thodi
> <skolothumtho@xxxxxxxxxx>; kevin.tian@xxxxxxxxx; bhelgaas@xxxxxxxxxx;
> dmatlack@xxxxxxxxxx; kees@xxxxxxxxxx; gustavoars@xxxxxxxxxx; Neo Jia
> <cjia@xxxxxxxxxx>; Krishnakant Jaju <kjaju@xxxxxxxxxx>; Vikram Sethi
> <vsethi@xxxxxxxxxx>; Zhi Wang <zhiw@xxxxxxxxxx>; linux-
> doc@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx;
> linux-cxl@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-
> kselftest@xxxxxxxxxxxxxxx; linux-hardening@xxxxxxxxxxxxxxx; alex@xxxxxxxxxxx
> Subject: Re: [PATCH v6 07/31] vfio/pci: Detect CXL devices and load the CXL
> provider on demand
>
> External email: Use caution opening links or attachments
>
>
> On Fri, 16 Oct 2026 21:48:14 +0530
> <mhonap@xxxxxxxxxx> wrote:
>
> > From: Manish Honap <mhonap@xxxxxxxxxx>
> >
> > A CXL device needs the vfio-cxl callbacks, but pulling vfio-cxl and
> > the CXL core in unconditionally would bloat every vfio-pci setup.
> > Detect a CXL Type-2 device at bind and request_module("vfio-cxl") only
> > then, then hand the device to the registered ops.
> >
> > Each bound CXL device pins the provider through
> > vfio_pci_get_cxl_ops() (try_module_get) and drops it with
> > vfio_pci_put_cxl_ops() at release, so vfio-cxl can unload once no CXL
> > device is bound.
> >
> > If the provider is absent the device is bound without the CXL
> > extensions. A built-in provider whose initcall has not run yet is
> > waited for with -EPROBE_DEFER; the deferred-probe machinery will retry
> > the bind once the provider registers.
> >
> > The provider .init() can return -EPROBE_DEFER when the device is not
> > ready yet, for example when its CXL port has not enumerated. Propagate
> > it so the bind is retried later. Any other init failure is non-fatal:
> > a CXL Type-2 device is also a valid vfio-pci device, so log the error,
> > drop the provider reference, leave cxl_ops NULL and continue without
> > CXL support.
> >
> > Assisted-by: LLM
> > Signed-off-by: Manish Honap <mhonap@xxxxxxxxxx>
> > ---
> > drivers/vfio/pci/vfio_pci_core.c | 101
> +++++++++++++++++++++++++++++++
> > include/linux/vfio_pci_core.h | 3 +
> > 2 files changed, 104 insertions(+)
> >
> > diff --git a/drivers/vfio/pci/vfio_pci_core.c
> > b/drivers/vfio/pci/vfio_pci_core.c
> > index 2cc5dd20396c..e50b36751d9a 100644
> > --- a/drivers/vfio/pci/vfio_pci_core.c
> > +++ b/drivers/vfio/pci/vfio_pci_core.c
> > @@ -2202,6 +2202,101 @@ void vfio_pci_core_unregister_cxl_ops(const
> > struct vfio_cxl_ops *ops) }
> > EXPORT_SYMBOL_GPL(vfio_pci_core_unregister_cxl_ops);
> >
> > +static const struct vfio_cxl_ops *vfio_pci_get_cxl_ops(void) {
> > + guard(rwsem_read)(&vfio_pci_cxl_ops_rwsem);
> > +
> > + if (vfio_pci_cxl_ops && try_module_get(vfio_pci_cxl_ops->owner))
> > + return vfio_pci_cxl_ops;
> > +
> > + return NULL;
> > +}
> > +
> > +static void vfio_pci_put_cxl_ops(const struct vfio_cxl_ops *ops) {
> > + module_put(ops->owner);
> > +}
> > +
> > +/*
> > + * A CXL Type-2 device advertises both CXL.cache and CXL.mem in its CXL
> DVSEC.
> > + * pcie_is_cxl() is also true for Type-1 (cache only) and Type-3 (mem
> > +only)
> > + * devices, which the vfio-cxl provider does not handle, so confirm
> > +the Type-2
> > + * identity before engaging it.
> > + */
>
> Probably another AI comment, it's only describing what the function is
> obviously doing. Drop it.
>

Okay, I will drop this.

> > +static bool vfio_pci_is_cxl_type2(struct pci_dev *pdev) {
> > + u16 dvsec, cap;
> > +
> > + if (!pcie_is_cxl(pdev))
> > + return false;
> > +
> > + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
> > + PCI_DVSEC_CXL_DEVICE);
> > + if (!dvsec)
> > + return false;
> > +
> > + if (pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP, &cap))
> > + return false;
> > +
> > + return (cap & PCI_DVSEC_CXL_CACHE_CAPABLE) &&
> > + (cap & PCI_DVSEC_CXL_MEM_CAPABLE); }
> > +
> > +/*
> > + * Load vfio-cxl on demand for a CXL Type-2 device and hand the
> > +device to its
> > + * ops. If the provider is absent the device is bound without the CXL
> > + * extensions; a built-in provider whose initcall has not run yet is
> > +waited for with
> > + * -EPROBE_DEFER.
> > + */
> > +static int vfio_pci_core_cxl_init(struct vfio_pci_core_device *vdev)
> > +{
> > + const struct vfio_cxl_ops *ops;
> > + int ret;
> > +
> > + if (!vfio_pci_is_cxl_type2(vdev->pdev))
> > + return 0;
> > +
> > + request_module("vfio-cxl");
>
> A local sashiko run questions:
> ---
> [Severity: High]
> Could request_module() in vfio_pci_core_cxl_init(), in
> drivers/vfio/pci/vfio_pci_core.c, deadlock when a matching Type-2 device is
> probed asynchronously and vfio-cxl is an unloaded module whose own
> async_probe setting remains false?
>
> Asynchronous probing can be selected with vfio-pci's async_probe=1 module
> argument or driver_async_probe=vfio-pci. The resulting wait cycle is:
>
> __driver_attach_async_helper()
> vfio_pci_core_cxl_init()
> request_module("vfio-cxl")
> waits for modprobe to exit
>
> do_init_module() for vfio-cxl
> async_synchronize_full()
> waits for the outstanding vfio-pci probe
>
> In kernel/module/main.c, do_init_module() performs this synchronization after
> the provider initializer returns. Successfully registering the operations therefore
> does not break the cycle. The provider's async_probe setting is independent of
> vfio-pci's setting.
>
> The warning in __request_module(), in kernel/module/kmod.c, does not abort
> loading. Dispatch through pci_call_probe(), in drivers/pci/pci-driver.c, also
> preserves the cycle because the originating async callback waits in
> flush_work(). The global async wait has no timeout.
>
> The final series retains this synchronous request. Could provider loading be
> moved outside the outstanding async probe, with binding deferred until the
> operations are available?
> ---
>
> AIUI, this could be resolved if we do the request_module() on bind and the CXL
> init on open, but CXL expects that we use the devm interfaces that have a bind-
> >release lifetime. But then...
>

Hmm, I will need to spend more time on this. With vfio-pci async_probe=1, I think
the probe blocks in request_module() while do_init_module() for vfio-cxl
waits in async_synchronize_full() for that same probe.

For v7, I will load vfio-cxl from vfio_pci_init() instead before
pci_register_driver():
- if a device present at that point has a CXL DVSEC that vfio-cxl can handle,
init calls request_module() and waits. Module init is not an async probe, so
the cycle suggested above cannot form, and a host without such a device will not
load CXL module, as per your suggestion on v3.

- A device that shows up while vfio-cxl is not loaded gets
request_module_nowait() and binds without CXL support, as it does
today when CXL setup fails. A rebind will pick it up.
I will use MODULE_WEAKDEP("vfio-cxl") to keep the module in the initramfs
without loading it.

> > + ops = vfio_pci_get_cxl_ops();
> > + if (!ops)
> > + return IS_BUILTIN(CONFIG_VFIO_CXL) ? -EPROBE_DEFER : 0;
> > +
> > + ret = ops->init(vdev);
> > + /*
> > + * A provider that is not ready yet (for example its CXL port has not
> > + * enumerated) returns -EPROBE_DEFER. Propagate it so the bind retries
> > + * rather than continuing without CXL support.
> > + */
> > + if (ret == -EPROBE_DEFER) {
> > + vfio_pci_put_cxl_ops(ops);
> > + return ret;
> > + }
> > + /*
> > + * Any other failure is non-fatal: a CXL device that cannot be brought
> > + * up as Type-2 still works without the CXL extensions, so log and
> > + * continue rather than failing the bind.
> > + */
> > + if (ret) {
> > + pci_warn(vdev->pdev,
> > + "CXL init failed (%d), continuing without CXL support\n",
> > + ret);
> > + vfio_pci_put_cxl_ops(ops);
> > + return 0;
> > + }
> > +
> > + vdev->cxl_ops = ops;
> > + return 0;
> > +}
> > +
> > +static void vfio_pci_core_cxl_release(struct vfio_pci_core_device
> > +*vdev) {
> > + if (!vdev->cxl_ops)
> > + return;
> > +
> > + vdev->cxl_ops->release(vdev);
> > + vfio_pci_put_cxl_ops(vdev->cxl_ops);
> > +}
> > +
> > int vfio_pci_core_init_dev(struct vfio_device *core_vdev) {
> > struct vfio_pci_core_device *vdev = @@ -2223,6 +2318,10 @@ int
> > vfio_pci_core_init_dev(struct vfio_device *core_vdev)
> > init_rwsem(&vdev->memory_lock);
> > xa_init(&vdev->ctx);
> >
> > + ret = vfio_pci_core_cxl_init(vdev);
> > + if (ret)
> > + return ret;
> > +
> > return 0;
> > }
> > EXPORT_SYMBOL_GPL(vfio_pci_core_init_dev);
> > @@ -2232,6 +2331,8 @@ void vfio_pci_core_release_dev(struct vfio_device
> *core_vdev)
> > struct vfio_pci_core_device *vdev =
> > container_of(core_vdev, struct vfio_pci_core_device,
> > vdev);
> >
> > + vfio_pci_core_cxl_release(vdev);
>
> Sashiko has a doozy regarding the resulting lifecycle teardown:
>
> ---
> [Severity: Critical]
> Can vfio_pci_core_release_dev(), in drivers/vfio/pci/vfio_pci_core.c, invoke
> provider teardown after device-managed cleanup has freed vdev->cxl when
> final object destruction is delayed during unbind?
>
> In the final series, vfio_cxl_create_memdev(), in
> drivers/vfio/pci/cxl/vfio_cxl_core.c, allocates the provider state through
> devm_cxl_dev_state_create() against pdev->dev. This release hook instead
> runs at final vfio object destruction, which can follow driver removal and
> device-managed cleanup.
>
> Device-cdev close and unbind can interleave as follows:
>
> vfio_device_fops_release()
> drops the last registration reference
> completes the unregister wait
>
> vfio_pci_remove()
> finishes unregistering
> drops its vfio device reference
> returns while the cdev still retains the vfio object
>
> device_unbind_cleanup()
> devres_release_all() frees the provider state
>
> __fput()
> cdev_put() drops the remaining parent reference
> vfio_pci_core_release_dev() calls provider teardown
>
> Enabling delayed kobject release provides another path without an open file.
> The module pin preserves the provider callback text, but does not preserve its
> device-managed state.
>
> vfio_cxl_release_device(), in drivers/vfio/pci/cxl/vfio_cxl_core.c, then reads
> the freed cxl->hpa_range and cxl->hdm to release the reservation and free the
> decoder snapshot. Reuse of that allocation can cause an incorrect reservation
> release or an invalid free. The final series retains both the managed allocation
> and this final-release hook.
>
> Could provider teardown run before device-managed cleanup, or could the
> provider state remain allocated through final vfio release?

I agree this is a bug in v6. I will handle this in v7. vfio-cxl state is
allocated with devm agains the PCI device and freed at unbind. But the teardown
happens at final vfio_device release. This can come later if cdev file
outlives the unbind.

For v7, I will setup CXL objects in vfio_pci_core_register_device and tear
them down in vfio_pci_core_unregister_device. This way, nothing CXL is left
in release_dev. CXL lifetime will be from open to close only.

I will make sure this is not introducing any bugs by running a local sashiko run.

Thanks,
Manish

> ---
>
> So perhaps trying to shoehorn CXL's devm lifecycle into vfio-pci is biting us
> anyway. Thanks,
>
> Alex