RE: [PATCH V3] PCI: Extend ACS configurability

[Vidya Sagar]

Hi Bjorn,
Could you let me know if Jason's reply answers your question?
Please let me know if you are looking for any more information.

Thanks,
Vidya Sagar

> -----Original Message-----
> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Sent: Thursday, May 23, 2024 8:46 PM
> To: Bjorn Helgaas <helgaas@xxxxxxxxxx>
> Cc: Vidya Sagar <vidyas@xxxxxxxxxx>; corbet@xxxxxxx; bhelgaas@xxxxxxxxxx; Gal
> Shalom <galshalom@xxxxxxxxxx>; Leon Romanovsky <leonro@xxxxxxxxxx>; Thierry
> Reding <treding@xxxxxxxxxx>; Jon Hunter <jonathanh@xxxxxxxxxx>; Masoud
> Moshref Javadi <mmoshrefjava@xxxxxxxxxx>; Shahaf Shuler <shahafs@xxxxxxxxxx>;
> Vikram Sethi <vsethi@xxxxxxxxxx>; Shanker Donthineni <sdonthineni@xxxxxxxxxx>;
> Jiandi An <jan@xxxxxxxxxx>; Tushar Dave <tdave@xxxxxxxxxx>; linux-
> doc@xxxxxxxxxxxxxxx; linux-pci@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> Krishna Thota <kthota@xxxxxxxxxx>; Manikanta Maddireddy
> <mmaddireddy@xxxxxxxxxx>; sagar.tv@xxxxxxxxx; Joerg Roedel <joro@xxxxxxxxxx>;
> Will Deacon <will@xxxxxxxxxx>; Robin Murphy <robin.murphy@xxxxxxx>;
> iommu@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH V3] PCI: Extend ACS configurability
> 
> On Thu, May 23, 2024 at 09:59:36AM -0500, Bjorn Helgaas wrote:
> > [+cc iommu folks]
> >
> > On Thu, May 23, 2024 at 12:05:28PM +0530, Vidya Sagar wrote:
> > > For iommu_groups to form correctly, the ACS settings in the PCIe
> > > fabric need to be setup early in the boot process, either via the
> > > BIOS or via the kernel disable_acs_redir parameter.
> >
> > Can you point to the iommu code that is involved here?  It sounds like
> > the iommu_groups are built at boot time and are immutable after that?
> 
> They are created when the struct device is plugged in. pci_device_group() does the
> logic.
> 
> Notably groups can't/don't change if details like ACS change after the groups are
> setup.
> 
> There are alot of instructions out there telling people to boot their servers and then
> manually change the ACS flags with set_pci or something, and these are not good
> instructions since it defeats the VFIO group based security mechanisms.
> 
> > If we need per-device ACS config that depends on the workload, it
> > seems kind of problematic to only be able to specify this at boot
> > time.  I guess we would need to reboot if we want to run a workload
> > that needs a different config?
> 
> Basically. The main difference I'd see is if the server is a VM host or running bare
> metal apps. You can get more efficicenty if you change things for the bare metal case,
> and often bare metal will want to turn the iommu off while a VM host often wants
> more of it turned on.
> 
> > Is this the iommu usage model we want in the long term?
> 
> There is some path to more dynamic behavior here, but it would require separating
> groups into two components - devices that are together because they are physically
> sharing translation (aliases and things) from devices that are together because they
> share a security boundary (ACS).
> 
> It is more believable we could dynamically change security group assigments for VFIO
> than translation group assignment. I don't know anyone interested in this right now -
> Alex and I have only talked about it as a possibility a while back.
> 
> FWIW I don't view patch as excluding more dynamisism in the future, but it is the best
> way to work with the current state of affairs, and definitely better than set_pci
> instructions.
> 
> Thanks,
> Jason