Re: [PATCH v2 3/5] iommu/amd: Introduce Disable IRTE Caching Support

[Alejandro Jimenez]

Hi Suravee,

On 5/18/2023 8:55 PM, Suravee Suthikulpanit wrote:
> An Interrupt Remapping Table (IRT) stores interrupt remapping configuration
> for each device. In a normal operation, the AMD IOMMU caches the table
> to optimize subsequent data accesses. This requires the IOMMU driver to
> invalidate IRT whenever it updates the table. The invalidation process
> includes issuing an INVALIDATE_INTERRUPT_TABLE command following by
> a COMPLETION_WAIT command.
>
> However, there are cases in which the IRT is updated at a high rate.
> For example, for IOMMU AVIC, the IRTE[IsRun] bit is updated on every
> vcpu scheduling (i.e. amd_iommu_update_ga()). On system with large
> amount of vcpus and VFIO PCI pass-through devices, the invalidation
> process could potentially become a performance bottleneck.
>
> Introducing a new kernel boot option:
>
>      amd_iommu=irtcachedis
>
> which disables IRTE caching by setting the IRTCachedis bit in each IOMMU
> Control register, and bypass the IRT invalidation process.
>
> Co-developed-by: Alejandro Jimenez <alejandro.j.jimenez@xxxxxxxxxx>
> [Awaiting sign-off by Alejandro]
Signed-off-by: Alejandro Jimenez <alejandro.j.jimenez@xxxxxxxxxx

I sanity tested the kdump mechanism, and confirmed that 
CONTROL_IRTCACHEDIS is set appropriately based on the irtcachedis kernel 
parameter.

Also, I have not observed any errors during multiple rounds of testing 
with large vCPU counts and VFIO passthrough devices under 
interrupt-intensive workload that causes heavy vCPU scheduling activity 
and exercises the relevant code path.

Thank you,
Alejandro
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@xxxxxxx>
> ---
>   .../admin-guide/kernel-parameters.txt         |  1 +
>   drivers/iommu/amd/amd_iommu_types.h           |  4 +++
>   drivers/iommu/amd/init.c                      | 36 +++++++++++++++++++
>   3 files changed, 41 insertions(+)