Re: [PATCH v2] iommu/arm-smmu-v3: Add a user-configurable tlb_invalidate_threshold

[Nicolin Chen]

On Fri, Aug 18, 2023 at 05:11:19PM +0100, Will Deacon wrote:
 
> On Thu, Aug 17, 2023 at 11:36:18AM -0700, Nicolin Chen wrote:
> > On Wed, Aug 16, 2023 at 01:43:50PM -0700, Nicolin Chen wrote:
> >
> > > When receiving an __arm_smmu_tlb_inv_range() call with a large size, there
> > > could be a long latency at this function call: one part is coming from a
> > > large software overhead in the routine of building commands, and the other
> > > part is coming from CMDQ hardware consuming the large number of commands.
> > > This latency could be significantly large on an SMMU that does not support
> > > range invalidation commands, i.e. no ARM_SMMU_FEAT_RANGE_INV.
> > >
> > > One way to optimize this is to replace a large number of VA invalidation
> > > commands with one single per-asid invalidation command, when the requested
> > > size reaches a threshold. This threshold can be configurable depending on
> > > the SMMU implementaion.
> >
> > I'm rethinking about this size-based threshold, since what really
> > affects the latency is the number of the invalidation commands in
> > the request. So having an npages-based threshold might be optimal,
> > though the idea and implementation would be similar.
> 
> On the CPU side, we just have:
> 
> #define MAX_TLBI_OPS    PTRS_PER_PTE
> 
> in asm/tlbflush.h
> 
> Can we start off with something similar for the SMMU? I'm not massively
> keen on exposing this as a knob to userspace, because I don't think most
> people will have a clue about how to tune it.

Yes! I was hesitating about an arbitrary threshold setup that actually
fits our situation better. Now it makes sense. Thanks for the input!

What I would do was to pick a number based on our test results. Yet, it
seems that 1024 was chosen at the beginning to fix a softlock bug, and
it changed to PTRS_PER_PTE for a better perf, IIUIC. Should we use the
similar setup for SMMU? I found it is stored in data->bits_per_level,
so perhaps pass it back to the driver via "struct io_pgtable_cfg".

Nicolin