RE: [RFC PATCH v4 03/16] iommu/arm-smmu-v3: Add initial pSMMU realm viommu plumbing

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxx>
> Sent: Thursday, September 10, 2026 8:47 PM
> 
> On Thu, Sep 10, 2026 at 09:52:12AM +0000, Tian, Kevin wrote:
> > > The mmio is owned by vfio, there should be a handshake in VFIO to
> > > remove mmio when it becomes private, otherwise I don't think we need
> > > to do anything more?
> >
> > for TDX a callback in VFIO is invoked to revoke the dmabuf.
> >
> > btw I didn't see any dmabuf related in this series (even in a hackish
> > way as Yilun sent out last year). Is it not required by CCA or just
> > skipped now waiting for the framework to be settled?
> 
> I think just one step at a time :\ I think CCA needs it as well, you
> also cannot safely leave private MMIO floating around the host
> userspace on CCA.

Good to know

> 
> > So viommu will become the abstraction point for tsm operations and
> > directly talks to the tsm driver instead of going through the merged
> > tsm_ops (bind/unbind/guest_req).
> 
> More precisely for what was being called the TDI - ie it is the
> abstraction point for the host side of a guest virtual PCI device.
> 
> iommufd vdevice already represents this, so we use it again.
> 
> > currently what tsm_ops provides additionally is more about synchronization
> > with connect/disconnect, otherwise just calls into underlying tsm driver. As
> > long as viommu creation holds a reference to tsm then disconnection is
> > blocked then it's safe.
> 
> Yes, this already has to be true or our lifecylce model is
> nonsense. Once the iommufd creates the vdevice the guest is running
> and we cannot disconnect it without unplugging it from the guest. Thus
> the locking can rely on that. Within an iommufd ioctl context the vdev
> and underlying connection must be stable.

btw looks a related suggestion is having the tsm driver (arm-cca-host
here) implement the viommu ops while leaving the original smmu driver
largely intact, and "obtain the viommu through tsm_ops not through
iommu_ops".

It's not super clear to me about the implication behind after reading
related discussions and what the "obtain" part means. 

Could you help elaborate this part?

The old way is kind of:

vdevice_tsm_ops -> tsm_ops -> tsm driver -> iommu driver

then it will become:

viommu_ops (tsm related) -> tsm_driver

where tsm driver will fully handle tsm related viommu_ops?

> 
> > and IOMMU_VDEVICE_TSM_BIND will be removed.
> 
> Replaced, the same logical operations should flow through the viommu
> command ioctl
> 
> > Seems originally this series
> > does SMC_RMI_VDEV_CREATE and SMC_RMI_VDEV_LOCK both in tsm
> > bind op. Then the proposal is moving them all to vdevice creation time so
> > no separate bind step is required.
> 
> Hmm, lock still should be guest visible on CCA, and the device has to
> start up in the guest as TDISP unlocked. I'm not 100% on the RMI side
> of the flow, but this does not look right.
> 
> VDEV create should be done before the realm is started. We need this
> setup right because Linux VM is going to validate the vdev during boot
> if it is affiliated to a vSMMU. lock should be done only when
> requested by the guest. The guest must start with a normal unlocked
> T=0 PCI device.

Okay then this part is aligned cross all vendors. Thanks for clarification!

> 
> > Does it mean that ARM CCA is essentially an early-bind model i.e. the TDI
> > starts in locked state from guest p.o.v.?
> 
> No
> 
> > but https://lore.kernel.org/all/yq5aecfbzjyk.fsf@xxxxxxxxxx/ seems to
> indicate
> > that guest_req() still handles the bind request from the guest:
> >
> > "
> > - cca_tsm_guest_req() now handles TSM_REQ_SET_TDI_STATE requests for
> >    the unlocked, locked, and running states.
> > "
> 
> Right the guest side request to lock should come that way.
> 
> >
> > a bit confusing, but maybe due to stale info cited from different timing...
> >
> > for TDX we're currently pursuing a late-bind model (i.e. TDI starts in shared
> > state in guest), so a separate bind interface is still necessary. We could look
> > at whether it should be an explicit @bind viommu op or carried by
> > @guest_req (say, if most preparation works can be moved to vdevice
> creation).
> 
> It should go over the viommu command ioctl. It looked like there was
> alway some iommu specific format to the command in the tsm version of
> this, so that is the appropriate way.
> 
> > btw per past discussions VFIO should block some operations (reset, etc.)
> > while a TDI is locked. Do we expect a callback into VFIO to notify the locked
> > state, or require VFIO's own interface to place a cdev into a special state
> > which prevents sensitive operations even before binding it to iommufd
> > (and allow creating TDI vdevice only on such idevice)?
> 
> At a minimum I think we have to synchronize with vfio, somehow, that
> when the device enforces its private MMIO so it can unmap it. Thay may
> be an argument we need a few standard viommu ops so the core code can
> capture lock/unlocks and do this prep work. Or maybe a special cdev
> state is simpler.
> 
> This should be figured out before any uapi is settled.

Agree

> 
> I don't know about reset, that sounds like something the pci core
> should deal with?
> 

yes the pci core sounds a better place to handle it.

btw I held the impression of blocked operations from past discussion [1].
Initial attempt tried to proactively unbind the TDI upon any operations 
which may transit the TDI to the ERROR state. Then the suggestion at
the moment was:

" But now the suggestion is never let VFIO do unbind, instead VFIO
should block these operations when device is bound. "

[1] https://lore.kernel.org/all/aEFmPaYorqaYCKBY@yilunxu-OptiPlex-7050/