Re: [PATCH v11 13/16] iommu: Improve iopf_queue_remove_device()

[Jason Gunthorpe]

On Mon, Feb 05, 2024 at 07:55:23PM +0800, Baolu Lu wrote:
> On 2024/2/5 17:00, Tian, Kevin wrote:
> > > From: Lu Baolu <baolu.lu@xxxxxxxxxxxxxxx>
> > > Sent: Tuesday, January 30, 2024 4:09 PM
> > >    *
> > > - * Caller makes sure that no more faults are reported for this device.
> > > + * Removing a device from an iopf_queue. It's recommended to follow
> > > these
> > > + * steps when removing a device:
> > >    *
> > > - * Return: 0 on success and <0 on error.
> > > + * - Disable new PRI reception: Turn off PRI generation in the IOMMU
> > > hardware
> > > + *   and flush any hardware page request queues. This should be done
> > > before
> > > + *   calling into this helper.
> > 
> > this 1st step is already not followed by intel-iommu driver. The Page
> > Request Enable (PRE) bit is set in the context entry when a device
> > is attached to the default domain and cleared only in
> > intel_iommu_release_device().
> > 
> > but iopf_queue_remove_device() is called when IOMMU_DEV_FEAT_IOPF
> > is disabled e.g. when idxd driver is unbound from the device.
> > 
> > so the order is already violated.
> > 
> > > + * - Acknowledge all outstanding PRQs to the device: Respond to all
> > > outstanding
> > > + *   page requests with IOMMU_PAGE_RESP_INVALID, indicating the device
> > > should
> > > + *   not retry. This helper function handles this.
> > > + * - Disable PRI on the device: After calling this helper, the caller could
> > > + *   then disable PRI on the device.
> > 
> > intel_iommu_disable_iopf() disables PRI cap before calling this helper.
> 
> You are right. The individual drivers should be adjusted accordingly in
> separated patches. Here we just define the expected behaviors of the
> individual iommu driver from the core's perspective.

Yeah, I don't think the driver really works properly before this
documentation was added either :\

We also need to check that the proposed AMD patches (SVA support part
4) are working right before they are merged.

Jason