RE: [PATCH v5 26/27] Documentation: vfio-pci: Document CXL Type-2 device passthrough

[Manish Honap]

> -----Original Message-----
> From: Jonathan Cameron <jic23@xxxxxxxxxx>
> Sent: Thursday, September 24, 2026 6:24 AM
> To: Manish Honap <mhonap@xxxxxxxxxx>
> Cc: Gregory Price <gourry@xxxxxxxxxx>; alex@xxxxxxxxxxx; jgg@xxxxxxxx;
> Ankit Agrawal <ankita@xxxxxxxxxx>; dave.jiang@xxxxxxxxx; alejandro.lucero-
> palau@xxxxxxx; Srirangan Madhavan <smadhavan@xxxxxxxxxx>;
> corbet@xxxxxxx; skhan@xxxxxxxxxxxxxxxxxxx; dave@xxxxxxxxxxxx;
> alison.schofield@xxxxxxxxx; vishal.l.verma@xxxxxxxxx; iweiny@xxxxxxxxxx;
> ming.li@xxxxxxxxxxxx; Yishai Hadas <yishaih@xxxxxxxxxx>; Shameer
> Kolothum Thodi <skolothumtho@xxxxxxxxxx>; kevin.tian@xxxxxxxxx;
> bhelgaas@xxxxxxxxxx; dmatlack@xxxxxxxxxx; kees@xxxxxxxxxx;
> gustavoars@xxxxxxxxxx; Neo Jia <cjia@xxxxxxxxxx>; Krishnakant Jaju
> <kjaju@xxxxxxxxxx>; Vikram Sethi <vsethi@xxxxxxxxxx>; Zhi Wang
> <zhiw@xxxxxxxxxx>; linux-doc@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> kvm@xxxxxxxxxxxxxxx; linux-cxl@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx;
> linux-kselftest@xxxxxxxxxxxxxxx; linux-hardening@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH v5 26/27] Documentation: vfio-pci: Document CXL Type-
> 2 device passthrough
> 
> External email: Use caution opening links or attachments
> 
> 
> On Mon, 21 Sep 2026 09:43:34 +0000
> Manish Honap <mhonap@xxxxxxxxxx> wrote:
> 
> > Hello Gregory,
> >
> > Thank you for the careful read of the documentation. I will rework the
> > sections you flagged as below:
> >
> 
> Hi Manish,
> 
> It is much more helpful to reply inline with the relevant sections so we can see
> the context wrt to what Gregory was asking.  That saves others reading the
> discussion from having to carefully build up a global model of what was said to
> understand what the replies are about.  Also I'll hazzard a guess that Gregory
> may have read enough code since he sent the review that he has already
> forgotten some of his own comments! (maybe that's just
> me!)  I appreciate that doesn't always work if you are going to address the
> comment via more radical restructuring.  Or in short, please don't top post!
> 
> Thanks,
> 
> Jonathan
> 

Understood, sorry for the churn. I have answered Gregory's points
inline in a reply to his original mail. I will keep replies inline from now.

> 
> > - Address model
> >
> >   I will update the wording to mention that VMM does know the GPA: it
> builds
> >   the guest's CFMWS window and so chooses the guest-physical range the
> device
> >   can inhabit. What the host owns is the HPA and the placement within it, not
> >   knowledge of the GPA.
> >   I will add the two models you described, fixed placement for an accelerator
> >   that needs exact physical placement versus no fixed placement for pooled
> or
> >   compressed memory, with a small HPA/GPA example, and note that the
> current
> >   series under discussion implements the fixed-offset case.
> >
> > - Virtual decoders
> >
> >   I will rename the The "Guest decoder and commit" section to "Virtual
> >   decoders" and add a short flow showing a guest vdecoder write being
> absorbed
> >   and a live read returning COMMITTED from the host-locked physical
> decoder.
> >
> > - "The kernel virtualizes ..."
> >
> >   I will update it to say vfio-pci-core (its config-space permission hooks).
> >
> > - guest IOAS, stage-2 mapping, and struct-page-less coherent memory
> >
> >   I will define these terms where first used and include that the range is
> >   handed to the device whole and never onlined as system RAM. I will also
> add
> >   details why a stale stage-2 mapping must not outlive the HDM window, and
> how
> >   the VMM rebuilds it.
> >
> > - Reset
> >
> >   I will reword this section. FLRs are virtualized so a guest reset can't
> >   inflict physical CXL.mem/decoder effects on the host rather than loose
> >   wording "must not take an FLR"
> >
> > I can share the revised text ahead of the v6 posting if that is easier
> > to review.
> >
> > Thanks,
> > Manish
> >
> > > -----Original Message-----
> > > From: Gregory Price <gourry@xxxxxxxxxx>
> > > Sent: Thursday, September 17, 2026 1:03 AM
> > > To: Manish Honap <mhonap@xxxxxxxxxx>
> > > Cc: alex@xxxxxxxxxxx; jgg@xxxxxxxx; Ankit Agrawal
> > > <ankita@xxxxxxxxxx>; jic23@xxxxxxxxxx; dave.jiang@xxxxxxxxx;
> > > alejandro.lucero-palau@xxxxxxx; Srirangan Madhavan
> > > <smadhavan@xxxxxxxxxx>; corbet@xxxxxxx; skhan@xxxxxxxxxxxxxxxxxxx;
> > > dave@xxxxxxxxxxxx; alison.schofield@xxxxxxxxx;
> > > vishal.l.verma@xxxxxxxxx; iweiny@xxxxxxxxxx; ming.li@xxxxxxxxxxxx;
> > > Yishai Hadas <yishaih@xxxxxxxxxx>; Shameer Kolothum Thodi
> > > <skolothumtho@xxxxxxxxxx>; kevin.tian@xxxxxxxxx;
> > > bhelgaas@xxxxxxxxxx; dmatlack@xxxxxxxxxx; kees@xxxxxxxxxx;
> > > gustavoars@xxxxxxxxxx; Neo Jia <cjia@xxxxxxxxxx>; Krishnakant Jaju
> > > <kjaju@xxxxxxxxxx>; Vikram Sethi <vsethi@xxxxxxxxxx>; Zhi Wang
> > > <zhiw@xxxxxxxxxx>; linux- doc@xxxxxxxxxxxxxxx;
> > > linux-kernel@xxxxxxxxxxxxxxx; kvm@xxxxxxxxxxxxxxx;
> > > linux-cxl@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-
> > > kselftest@xxxxxxxxxxxxxxx; linux-hardening@xxxxxxxxxxxxxxx
> > > Subject: Re: [PATCH v5 26/27] Documentation: vfio-pci: Document CXL
> > > Type-
> > > 2 device passthrough
> > >
> > > External email: Use caution opening links or attachments
> > >
> > >
> > > On Thu, Sep 17, 2026 at 12:05:39AM +0530, mhonap@xxxxxxxxxx wrote:
> > > > From: Manish Honap <mhonap@xxxxxxxxxx>
> > > >
> > >
> > > 1) Thank you so much for writing documentation, i truly appreciate this.
> > >
> > > 2) I apologize in advance for my terseness, I know writing is hard,
> > >    please do not interpret this as disliking your writing or series.
> > >
> > > > +Address model
> > > > +=============
> > > > +
> > > > +The HDM memory is a coherent host physical range (HPA). The host
> > > > +kernel resolves that range before the guest sees the device, and
> > > > +owns it for the bind lifetime. The guest only chooses where the
> > > > +memory appears in its own physical address space (GPA), by
> > > > +programming a virtual endpoint HDM decoder. The guest never
> > > > +reprograms the physical
> > > decoder.
> > > > +
> > > > +The kernel holds the HPA and does not see the GPA. The guest
> > > > +programs a GPA and does not see the HPA. The VMM holds the device
> > > > +fd, reads the committed base from the decoder-register region
> > > > +described below, and maps the HPA-backed HDM region at the GPA the
> guest committed.
> > > > +The base the guest reads back is the GPA, not the HPA.
> > > > +
> > >
> > > I think this must be slightly inaccurate / imprecise wording.
> > >
> > > The host *must* provide some form of physical memory window to the
> > > guest at initialization time, otherwise the guest has no way to know
> > > - at boot time - that there's even a window of memory it can use.
> > >
> > > That's what the CFMWS is.  This is initialized by the hypervisor -
> > > which is controlled by the host.
> > >
> > > So the host (at least the VMM) must know, for the region the entire
> > > device
> > > *could* inhabit, what that GPA is - because it's the one that makes
> > > the CFMWS for the guest.
> > >
> > > If this is not the case, then something is missing from this
> > > documentation to explain why.
> > >
> > >
> > > If you're actually trying to say is that the GPA's programmed into
> > > the virtual decoders are largely symbolic - this at best feels a bit
> > > inaccurate and simply an implementation detail.
> > >
> > > The host's virtio device could enforce ....:
> > >
> > > Host Range:
> > >    CFMWS HPA - [0x10000, 0x20000]
> > >                    |         |
> > > Guest Range:       |         |
> > >    CFMWS GPA - [0x50000, 0x60000]
> > >
> > > In that case, you'd get the following translation...
> > >    vdecoder0.0 - [0x58000, 0x60000]
> > >    CFMWS GPA   - [0x58000, 0x60000]
> > >    CFMWS HPA   - [0x18000, 0x20000]
> > >
> > > Or the virtio device could not enforce that and let the host
> > > page-fault just hand it a random page from the actual CXL device.
> > >
> > >    vdecoder0.0 - [0x58000, 0x60000]
> > >    CFMWS GPA   - [0x58000, 0x60000]
> > >                          |
> > >               No discrete host mapping
> > >
> > >
> > > The former makes sense if the device (accelerator) requires exact
> > > physical placement to do its accelerator nonsense.
> > >
> > > The latter makes sense if the device (accelerator) doesn't care
> > > about placement (compressed memory).
> > >
> > > This is not saying we need support both out of the box, but we
> > > shouldn't lock ourselves into the former unless there's some reason
> > > why the latter is not reasonable.
> > >
> > > Can you please help document what the actual expected behavior is
> > > with examples in the Address model section so it's easier to understand the
> intent?
> > > That will help quite a bit.
> > >
> > > > +Guest decoder and commit
> > > > +========================
> > > > +
> > > > +The guest programs its virtual endpoint decoder through the
> > > > +trapped
> > > > +region: it writes a base (a GPA), a size, and then the COMMIT bit.
> > > > +The host already resolved and committed the physical placement
> > > > +before the guest ran,
> > >
> > > So the host does know GPA, just not exact placement.
> > >
> > > > so a live read of the decoder always shows COMMITTED and the
> > > > +guest's commit poll completes. The physical decoder is never
> > > > +rewritten; the guest's writes are absorbed.
> > > > +
> > >
> > > Rather clunky, round-about way to say "The guest decoders are
> > > virtualized".   If possible, it would be nice to formalize this concept
> > > into "Virtual Decoders" - since that's what this is.
> > >
> > > With that concept i think you can probably generate some nice
> > > diagrams that show how the guest vdecoder's interact with the host
> drivers.
> > >
> > > > +The VMM observes the commit, reads the committed base, and maps
> > > > +the HDM region at that GPA.
> > > > +
> > >
> > > So the host does know the GPA.
> > >
> > > > +CXL Device DVSEC
> > > > +================
> > > > +
> > > > +The kernel virtualizes the CXL Device DVSEC body through the
> > > > +config-space permission hooks. Reads and writes inside the DVSEC
> > > > +body use a per-open shadow; a guest write stays in the shadow and
> > > > +does not
> > > reach hardware.
> > > > +Accesses outside the DVSEC body go to the device as usual.
> > > > +
> > >
> > > "The kernel" - what part? vfio-pci ? the vmm ?
> > >
> > >
> > > > +DMA and iommufd
> > > > +===============
> > > > +
> > > > +A Type-2 accelerator issues ATS-translated DMA to addresses
> > > > +inside its own HDM window, so that range must be present in the
> > > > +guest IOAS that backs the nested stage-2 translation.
> > >
> > > Type-2, ATS, DMA, HDM window, guest IOAS, stage-2 translation
> > >
> > > I think the only thing i don't know in this sentence is "guest IOAS"
> > > and it's still hurting my brain to read.
> > >
> > > Are all accelerators expect to have this particular interaction, or just yours?
> > >
> > > > The HDM range is struct-page-less coherent
> > > > +memory, which a userspace-VA ``IOMMU_IOAS_MAP`` cannot pin.
> > > > +
> > >
> > > The hardest part about writing about virtualization is keeping a
> > > consistent mental model from section to section.
> > >
> > > which userspace? guest? host?  (i presume guest here)
> > >
> > > `struct-page-less coherent memory`
> > >    e.g. the host never hotplugs this, it hands the entire region
> > >    directly to the VFIO device, right?
> > >
> > >    I think this would be nice to spell out somewhere.
> > >
> > > > +The HDM memory region is therefore exportable as a dma-buf:
> > > > +``VFIO_DEVICE_FEATURE_DMA_BUF`` on that region returns an fd that
> > > > +iommufd maps with ``IOMMU_IOAS_MAP_FILE``, mapping the physical
> > > range
> > > > +without a VA or a page pin. The dma-buf is revoked whenever the
> > > > +mapping is torn down (reset, power transition, teardown), so a
> > > > +stale
> > > > +stage-2 mapping cannot outlive the HDM window.
> > > > +
> > >
> > > For the sake of readers, I think either a little bit more
> > > information on this
> > > "stage-2 mapping" concept is needed to make sense of what's going on
> > > here and why it mustn't outlive the HDM window.
> > >
> > > > +Reset
> > > > +=====
> > > > +
> > > > +A CXL Type-2 function must not take a Function Level Reset: an
> > > > +FLR resets the coherent CXL.mem state and the HDM decoder. The
> > > > +PCI core reflects this by preferring the CXL reset over FLR, so a
> > > > +function reset of a CXL device runs the CXL DVSEC reset sequence,
> > > > +which resets the function and then restores the HDM decoder and the
> PCI config state.
> > > > +
> > >
> > > I think what you're trying to say is that FLRs are never passed to
> > > the device because it can cause physical device effects that defeat
> > > the purpose of the virtualization, yes?
> > >
> > > So we virtualize FLRs...
> > >
> > > > +A guest requests a reset by writing Initiate CXL Reset in the DVSEC.
> > > > +That write only stamps completion in the shadow. The real reset
> > > > +runs at the vfio reset points (the reset ioctl and a virtualized
> > > > +FLR through config
> > > space):
> > >
> > > As you describe here.
> > >
> > > So it's not that an accelerator "must not take an FLR" - it's that
> > > FLRs are virtualized to prevent deleterious effects on the host / hardware.
> > >
> > > Am I misunderstanding this?
> > >
> > > > +the kernel zaps the HDM mapping and revokes the dma-buf, then
> > > > +runs the CXL reset, which always clears the device memory, and
> > > > +restores and re-samples the decoder afterwards. A CXL port masks
> > > > +Secondary Bus Reset by default, so a
> > > > +``VFIO_DEVICE_PCI_HOT_RESET`` does not reach the endpoint and the
> > > > +HDM state is untouched. If the port has SBR unmasked the reset
> > > > +can decommit the decoder without restoring it, so the reset_done
> > > > +handler gates HDM access; a ``VFIO_DEVICE_RESET`` then runs the CXL
> reset sequence and restores it.
> > > > +
> > > > +The decoder register region is served by live reads of the
> > > > +hardware decoder with guest writes absorbed: the decoder is
> > > > +committed and locked by the host, so a guest can neither decommit
> > > > +nor reprogram it, and the kernel keeps no shadow of the decoder state.
> > >
> > > This is basically what I said at the beginning - it must either be
> > > that the host provides locked auto-decoders at boot, or it must
> > > provide proper virtualization so that the decoders settings are fully
> virtualized.
> > >
> > > Seems it's the former, and that makes sense.  Please correct me if
> > > i'm misunderstanding.
> > >
> > > > After a reset the kernel restores and
> > > > +re-samples the firmware-committed decoder, so the geometry the
> > > > +guest reads back is unchanged. A VMM that dropped its HDM
> > > > +mapping, for example across a reset or a D3hot->D0 transition,
> > > > +must rescan the decoder and rebuild its
> > > > +stage-2 mapping before it resumes HDM access.
> > > > +
> > >
> > > Yeah i think we need a bit more information about this stage-2
> > > mapping rebuild to make sense of this.  Maybe I'm just not read-up
> > > enough on this particular setup - is there another part of the docs
> > > you can link to that talk about this, or are you able to share some
> > > details as to what this rebuild process looks like?
> > >
> > > ~Gregory