RE: [PATCH v6 11/31] vfio/pci: Virtualize the CXL DVSEC in vfio_pci_config.c

From: Manish Honap

Date: Wed Nov 18 2026 - 14:24:58 EST




> -----Original Message-----
> From: Alex Williamson <alex@xxxxxxxxxxx>
> Sent: Saturday, November 7, 2026 3:41 AM
> To: Manish Honap <mhonap@xxxxxxxxxx>
> Cc: jgg@xxxxxxxx; Ankit Agrawal <ankita@xxxxxxxxxx>; jic23@xxxxxxxxxx;
> dave.jiang@xxxxxxxxx; alejandro.lucero-palau@xxxxxxx; Srirangan Madhavan
> <smadhavan@xxxxxxxxxx>; corbet@xxxxxxx; skhan@xxxxxxxxxxxxxxxxxxx;
> dave@xxxxxxxxxxxx; alison.schofield@xxxxxxxxx; vishal.l.verma@xxxxxxxxx;
> iweiny@xxxxxxxxxx; ming.li@xxxxxxxxxxxx; Yishai Hadas
> <yishaih@xxxxxxxxxx>; Shameer Kolothum Thodi
> <skolothumtho@xxxxxxxxxx>; kevin.tian@xxxxxxxxx; bhelgaas@xxxxxxxxxx;
> dmatlack@xxxxxxxxxx; kees@xxxxxxxxxx; gustavoars@xxxxxxxxxx; Neo Jia
> <cjia@xxxxxxxxxx>; Krishnakant Jaju <kjaju@xxxxxxxxxx>; Vikram Sethi
> <vsethi@xxxxxxxxxx>; Zhi Wang <zhiw@xxxxxxxxxx>; linux-
> doc@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx;
> linux-cxl@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-
> kselftest@xxxxxxxxxxxxxxx; linux-hardening@xxxxxxxxxxxxxxx; alex@xxxxxxxxxxx
> Subject: Re: [PATCH v6 11/31] vfio/pci: Virtualize the CXL DVSEC in
> vfio_pci_config.c
>
> External email: Use caution opening links or attachments
>
>
> On Fri, 16 Oct 2026 21:48:18 +0530
> <mhonap@xxxxxxxxxx> wrote:
>
> > From: Manish Honap <mhonap@xxxxxxxxxx>
> >
> > A CXL Type-2 device is reprogrammable through its CXL DVSEC: a guest
> > could set Config Lock, toggle CXL.cache and CXL.mem enable, or rewrite
> > the HDM range registers that govern host memory decode. Virtualize the
> > DVSEC so the guest sees a shadow it cannot use to reprogram the hardware.
> >
> > Build a per-device cxl_perm permission map, modeled on msi_perm. When
> > the device is bound through the CXL provider (e.g. vfio-cxl), the
> > whole CXL DVSEC is served from the vconfig shadow; only Control and
> > Control2 are guest programmable, while Capability, Status, Lock and
> > the Range registers keep their firmware snapshot. A vendor DVSEC on
> > the same device is unaffected: the map is selected by the CXL DVSEC's
> > range, since every DVSEC has the same ID in the config map and two
> > DVSECs can be adjacent.
> >
> > A Type-2 function bound without the provider (the provider is absent,
> > its init failed, or the extensions are disabled) gets a guard map
> > instead. The DVSEC reads from hardware, Control and Control2 stay
> > writable and the RW1C status bits stay clearable, but writes to Lock
> > and the Range registers are dropped, since they outlive the user.
> > Initiate CXL Reset uses the same doorbell; a cache write-back goes to the
> device.
>
> The guard map actually changes the device ABI. This may be necessary, but if
> so, it should be pulled out into a Fixes: patch. The device aspects that change
> need to be well justified. Access to CXL reset without coordination is certainly
> not good. Config lock has side effects that we may not want to deal with. Is
> there value in exposing the CXL DVSEC at all when we're operating in vfio-pci
> mode? What do we expect a type-2 driver does with the guarded capability?
>
> Maybe the right answer is to drop it from the chain and make it read-only.
>

okay, for v7 I will drop the guard map and make the CXL DVSEC of a Type-2
function read-only when vfio-cxl is not bound, as a separate patch at
the front of the series with appropriate fixes: tag.

> > Control2 carries the CXL reset and cache write-back-invalidate
> > initiate bits as self-clearing doorbells. With the provider, vfio
> > never forwards them to hardware, so a custom writefn synthesizes their
> > completion in the shadow: the initiate bit self-clears and the
> > matching Status2 bit (Cache Invalid or Reset Done) is set, so a guest
> > that writes back its cache (CXL r4.0 sec 9.6) and then runs CXL Reset
> > (sec 9.7.2) progresses instead of timing out. Cache Invalid is
> > reported only while Disable Caching is set, as the spec requires. The
> > host performs the real cache write-back and reset at the vfio reset
> > points. Until a later patch in this series runs the real reset from
> > the doorbell, Initiate CXL Reset completes in the shadow only.
> >
> > Assisted-by: LLM
> > Signed-off-by: Manish Honap <mhonap@xxxxxxxxxx>
> > ---
> > drivers/vfio/pci/vfio_pci_config.c | 206 ++++++++++++++++++++++++++++-
> > include/linux/vfio_pci_core.h | 3 +
> > 2 files changed, 205 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/vfio/pci/vfio_pci_config.c
> > b/drivers/vfio/pci/vfio_pci_config.c
> > index 9914f3ac69ae..1fc463540211 100644
> > --- a/drivers/vfio/pci/vfio_pci_config.c
> > +++ b/drivers/vfio/pci/vfio_pci_config.c
> > @@ -1747,6 +1747,194 @@ static const struct pci_device_id
> known_bogus_vf_intx_pin[] = {
> > * for each area requiring emulated bits, but the array of pointers
> > * would be comparable in size (at least for standard config space).
> > */
>
> This comment above used to be attached to vfio_config_init() and now it's
> miles away!
>

I will update this.

> > +static int vfio_cxl_dvsec_write(struct vfio_pci_core_device *vdev, int pos,
> > + int count, struct perm_bits *perm,
> > + int offset, __le32 val) {
> > + u16 start = vdev->cxl_dvsec;
> > + __le16 *pctrl2, *pstatus2;
> > + u16 ctrl2, status2;
> > +
> > + count = vfio_default_config_write(vdev, pos, count, perm, offset, val);
> > + if (count < 0)
> > + return count;
> > +
> > + /* Only Control2 carries the self-clearing doorbells. */
> > + if (offset > PCI_DVSEC_CXL_CTRL2 ||
> > + offset + count <= PCI_DVSEC_CXL_CTRL2)
> > + return count;
> > +
> > + pctrl2 = (__le16 *)&vdev->vconfig[start + PCI_DVSEC_CXL_CTRL2];
> > + pstatus2 = (__le16 *)&vdev->vconfig[start + PCI_DVSEC_CXL_STATUS2];
> > + ctrl2 = le16_to_cpu(*pctrl2);
> > + status2 = le16_to_cpu(*pstatus2);
> > +
> > + /*
> > + * INIT_CACHE_WBI and INIT_CXL_RST are self-clearing doorbells. With a
> > + * provider neither reaches hardware: synthesize their completion in the
> > + * shadow so the guest poll finishes, clearing the initiate bit and
> > + * setting the matching Status2 completion. Cache Invalid is set only
> > + * while caching is disabled and reads 0 otherwise (CXL r4.0 sec 8.1.3.5,
> > + * Table 8-9). Under the guard map INIT_CACHE_WBI and Cache Invalid
> go to
> > + * the device; only INIT_CXL_RST and its completion are virtualized.
> > + */
> > + if (ctrl2 & PCI_DVSEC_CXL_INIT_CACHE_WBI) {
> > + ctrl2 &= ~PCI_DVSEC_CXL_INIT_CACHE_WBI;
> > + if (ctrl2 & PCI_DVSEC_CXL_DISABLE_CACHING)
> > + status2 |= PCI_DVSEC_CXL_CACHE_INV;
> > + }
> > + if (!(ctrl2 & PCI_DVSEC_CXL_DISABLE_CACHING))
> > + status2 &= ~PCI_DVSEC_CXL_CACHE_INV;
> > + if (ctrl2 & PCI_DVSEC_CXL_INIT_CXL_RST) {
> > + ctrl2 &= ~PCI_DVSEC_CXL_INIT_CXL_RST;
> > + status2 &= ~PCI_DVSEC_CXL_RST_ERR;
> > + status2 |= PCI_DVSEC_CXL_RST_DONE;
> > + }
> > +
> > + *pctrl2 = cpu_to_le16(ctrl2);
> > + *pstatus2 = cpu_to_le16(status2);
> > +
> > + return count;
> > +}
> > +
> > +static int init_cxl_dvsec_perm(struct perm_bits *perm, int len) {
> > + if (alloc_perm_bits(perm, len))
> > + return -ENOMEM;
> > +
> > + perm->writefn = vfio_cxl_dvsec_write;
> > +
> > + /* Serve the whole CXL DVSEC from the shadow. */
> > + for (int i = 0; i < len; i++)
> > + p_setb(perm, i, (u8)ALL_VIRT, NO_WRITE);
> > +
> > + /*
> > + * Control and Control2 are guest programmable; Capability, Status,
> > + * Lock and the Range registers keep their firmware snapshot, so the
> > + * guest cannot set Config Lock or rewrite the capability and ranges.
> > + */
> > + p_setw(perm, PCI_DVSEC_CXL_CTRL, (u16)ALL_VIRT, (u16)ALL_WRITE);
> > + p_setw(perm, PCI_DVSEC_CXL_CTRL2, (u16)ALL_VIRT,
> > + (u16)ALL_WRITE);
> > +
> > + return 0;
> > +}
> > +
> > +/*
> > + * Without a CXL provider the device is bound without the CXL
> > +extensions, so
> > + * the DVSEC is read from hardware, Control and Control2 stay
> > +writable, and
> > + * the RW1C status bits in Status and Status2 (for example
> > +Viral_Status) can
> > + * still be cleared. Drop the writes whose effect outlives the user:
> > +Config
> > + * Lock and the Range registers. Initiate CXL Reset goes through the
> > +same
> > + * doorbell as with a provider, so the reset runs at a vfio reset
> > +point under
> > + * memory_lock.
> > + */
> > +static int init_cxl_dvsec_guard_perm(struct perm_bits *perm, int len)
> > +{
> > + if (alloc_perm_bits(perm, len))
> > + return -ENOMEM;
> > +
> > + perm->writefn = vfio_cxl_dvsec_write;
> > +
> > + /* Extended capability header: ID and next pointer from vconfig. */
> > + p_setd(perm, 0, ALL_VIRT, NO_WRITE);
> > +
> > + p_setw(perm, PCI_DVSEC_CXL_CTRL, NO_VIRT, (u16)ALL_WRITE);
> > + p_setw(perm, PCI_DVSEC_CXL_CTRL2, PCI_DVSEC_CXL_INIT_CXL_RST,
> > + (u16)ALL_WRITE);
> > + /* Status (offset 0Eh, between Control and Control2): RW1C to the
> device. */
> > + p_setw(perm, PCI_DVSEC_CXL_CTRL + 2, NO_VIRT, (u16)ALL_WRITE);
> > + p_setw(perm, PCI_DVSEC_CXL_STATUS2,
> > + PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR,
> > + (u16)~(PCI_DVSEC_CXL_RST_DONE | PCI_DVSEC_CXL_RST_ERR));
> > +
> > + return 0;
> > +}
> > +
> > +/* Virtualize the CXL DVSEC so a guest cannot reprogram the device
> > +through it. */ static int vfio_cxl_dvsec_init(struct
> > +vfio_pci_core_device *vdev) {
> > + struct pci_dev *pdev = vdev->pdev;
> > + u32 dword;
> > + u16 dvsec, cap;
> > + int len, ret;
> > +
> > + dvsec = pci_find_dvsec_capability(pdev, PCI_VENDOR_ID_CXL,
> > + PCI_DVSEC_CXL_DEVICE);
> > + if (!dvsec)
> > + return 0;
> > +
> > + ret = pci_read_config_dword(pdev, dvsec + PCI_DVSEC_HEADER1,
> &dword);
> > + if (ret)
> > + return pcibios_err_to_errno(ret);
> > + len = PCI_DVSEC_HEADER1_LEN(dword);
> > +
> > + /*
> > + * The virtualization writes fixed DVSEC offsets up to Status2 (the reset
> > + * doorbell stamps it). A device that reports a shorter DVSEC is not a
> > + * usable Type-2 function; leave its DVSEC unvirtualized rather than
> > + * index the device-length-sized perm allocation past its end.
> > + */
> > + if (len < PCI_DVSEC_CXL_STATUS2 + 2)
> > + return 0;
> > +
> > + /*
> > + * Without a provider, guard only a CXL Type-2 function, with the same
> > + * test vfio-pci-core uses to load the provider.
> > + */
>
> Cryptic comment. Why don't we expose vfio_pci_is_cxl_type2() and use it
> here?

Okay, I will refactor this part.

>
> > + if (!vdev->cxl_ops) {
> > + if (!pcie_is_cxl(pdev))
> > + return 0;
> > + ret = pci_read_config_word(pdev, dvsec + PCI_DVSEC_CXL_CAP,
> &cap);
> > + if (ret)
> > + return pcibios_err_to_errno(ret);
> > + if (!(cap & PCI_DVSEC_CXL_CACHE_CAPABLE) ||
> > + !(cap & PCI_DVSEC_CXL_MEM_CAPABLE))
> > + return 0;
> > + }
> > +
> > + vdev->cxl_perm = kmalloc_obj(*vdev->cxl_perm,
> GFP_KERNEL_ACCOUNT);
> > + if (!vdev->cxl_perm)
> > + return -ENOMEM;
> > +
> > + if (vdev->cxl_ops)
> > + ret = init_cxl_dvsec_perm(vdev->cxl_perm, len);
> > + else
> > + ret = init_cxl_dvsec_guard_perm(vdev->cxl_perm, len);
> > + if (ret) {
> > + kfree(vdev->cxl_perm);
> > + vdev->cxl_perm = NULL;
> > + return ret;
> > + }
> > +
> > + vdev->cxl_dvsec = dvsec;
> > + vdev->cxl_dvsec_len = len;
> > +
> > + return 0;
> > +}
> > +
> > +static void vfio_cxl_dvsec_exit(struct vfio_pci_core_device *vdev) {
> > + if (!vdev->cxl_perm)
> > + return;
> > +
> > + free_perm_bits(vdev->cxl_perm);
> > + kfree(vdev->cxl_perm);
> > + vdev->cxl_perm = NULL;
> > + vdev->cxl_dvsec = 0;
> > + vdev->cxl_dvsec_len = 0;
> > +}
> > +
> > +/*
> > + * Every DVSEC carries the same ID in pci_config_map, so
> > +vfio_find_cap_start()
> > + * can walk from one DVSEC into an adjacent one. Match the CXL DVSEC by
> range.
> > + */
>
> IMO, a comment is more relevant below.

Okay, I will update this.

>
> > +static bool vfio_cxl_dvsec_contains(struct vfio_pci_core_device *vdev,
> > + loff_t pos) {
> > + return vdev->cxl_perm && pos >= vdev->cxl_dvsec &&
> > + pos < vdev->cxl_dvsec + vdev->cxl_dvsec_len; }
> > +
> > int vfio_config_init(struct vfio_pci_core_device *vdev) {
> > struct pci_dev *pdev = vdev->pdev; @@ -1842,6 +2030,10 @@ int
> > vfio_config_init(struct vfio_pci_core_device *vdev)
> > if (ret)
> > goto out;
> >
> > + ret = vfio_cxl_dvsec_init(vdev);
> > + if (ret)
> > + goto out;
> > +
> > return 0;
> >
> > out:
> > @@ -1863,6 +2055,7 @@ void vfio_config_free(struct vfio_pci_core_device
> *vdev)
> > kfree(vdev->msi_perm);
> > vdev->msi_perm = NULL;
> > }
> > + vfio_cxl_dvsec_exit(vdev);
>
> Teardown in opposite order of init. Last initialized ~ first exit.

Yes, I will fix this and use the comment at the cap_start call.

Thanks,
Manish

>
> > }
> >
> > /*
> > @@ -1926,12 +2119,17 @@ ssize_t vfio_pci_config_rw_single(struct
> vfio_pci_core_device *vdev,
> > * of the extended capability list. Use default, ro
> > * access, which will virtualize the id and next values.
> > */
> > - if (cap_id > PCI_EXT_CAP_ID_MAX)
> > + cap_start = vfio_find_cap_start(vdev, *ppos);
> > +
> > + if (cap_id > PCI_EXT_CAP_ID_MAX) {
> > perm = &direct_ro_perms;
> > - else
> > + } else if (cap_id == PCI_EXT_CAP_ID_DVSEC &&
> > + vfio_cxl_dvsec_contains(vdev, *ppos)) {
> > + perm = vdev->cxl_perm;
>
> /*
> * Multiple DVSECs may be back-to-back, resulting
> * in bogus cap_start.
> */
>
> Thanks,
> Alex
>
> > + cap_start = vdev->cxl_dvsec;
> > + } else {
> > perm = &ecap_perms[cap_id];
> > -
> > - cap_start = vfio_find_cap_start(vdev, *ppos);
> > + }
> > } else {
> > WARN_ON(cap_id > PCI_CAP_ID_MAX);
> >
> > diff --git a/include/linux/vfio_pci_core.h
> > b/include/linux/vfio_pci_core.h index 70acb03d466e..fd1ad0baf470
> > 100644
> > --- a/include/linux/vfio_pci_core.h
> > +++ b/include/linux/vfio_pci_core.h
> > @@ -120,6 +120,9 @@ struct vfio_pci_core_device {
> > u8 *pci_config_map;
> > u8 *vconfig;
> > struct perm_bits *msi_perm;
> > + struct perm_bits *cxl_perm;
> > + u16 cxl_dvsec;
> > + u16 cxl_dvsec_len;
> > spinlock_t irqlock;
> > struct mutex igate;
> > struct xarray ctx;