Re: [PATCH 1/1] iommu/arm-smmu: Fix for ThunderX erratum #27704

[Tomasz Nowicki]

On 12.01.2017 07:41, Tomasz Nowicki wrote:
> On 11.01.2017 13:19, Robin Murphy wrote:
> > On 11/01/17 11:51, Tomasz Nowicki wrote:
> > > The goal of erratum #27704 workaround was to make sure that ASIDs and
> > > VMIDs
> > > are unique across all SMMU instances on affected Cavium systems.
> > >
> > > Currently, the workaround code partitions ASIDs and VMIDs by increasing
> > > global cavium_smmu_context_count which in turn becomes the base ASID
> > > and VMID
> > > value for the given SMMU instance upon the context bank initialization.
> > >
> > > For systems with multiple SMMU instances this approach implies the risk
> > > of crossing 8-bit ASID, like for CN88xx capable of 4 SMMUv2, 128
> > > context bank each:
> > > SMMU_0 (0-127 ASID RANGE)
> > > SMMU_1 (127-255 ASID RANGE)
> > > SMMU_2 (256-383 ASID RANGE) <--- crossing 8-bit ASID
> > > SMMU_3 (384-511 ASID RANGE) <--- crossing 8-bit ASID
> >
> > I could swear that at some point in the original discussion it was said
> > that the TLBs were only shared between pairs of SMMUs, so in fact 0/1
> > and 2/3 are independent of each other
>
> Indeed TLBs are only shared between pairs of SMMUs but the workaround
> makes sure ASIDs are unique across all SMMU instances so we do not have
> to bother about SMMUs probe order.
>
>  - out of interest, have you
> > managed to hit an actual problem in practice or is this patch just by
> > inspection?
>
> Except SMMU0/1 devices all other devices under other SMMUs will fail on
> guest power off/on. Since we try to invalidate tlb with 16bit ASID but
> we actually have 8 bit zero padded 16 bit entry.
>
> > Of course, depending on the SMMUs to probe in the right order isn't
> > particularly robust, so it's still probably a worthwhile change.
> >
> > > Since we use 8-bit ASID now we effectively misconfigure ASID[15:8]
> > > bits for
> > > SMMU_CBn_TTBRm register. Also, we still use non-zero ASID[15:8] bits
> > > upon context invalidation. This patch adds 16-bit ASID support for
> > > stage-1
> > > AArch64 contexts for Cavium SMMUv2 model so that we use ASIDs
> > > consistently.
> > >
> > > Signed-off-by: Tomasz Nowicki <tn@xxxxxxxxxxxx>
> > > ---
> > >  drivers/iommu/arm-smmu.c | 4 ++++
> > >  1 file changed, 4 insertions(+)
> > >
> > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c
> > > index a60cded..ae8f059 100644
> > > --- a/drivers/iommu/arm-smmu.c
> > > +++ b/drivers/iommu/arm-smmu.c
> > > @@ -260,6 +260,7 @@ enum arm_smmu_s2cr_privcfg {
> > >
> > >  #define TTBCR2_SEP_SHIFT        15
> > >  #define TTBCR2_SEP_UPSTREAM        (0x7 << TTBCR2_SEP_SHIFT)
> > > +#define TTBCR2_AS            (1 << 4)
> > >
> > >  #define TTBRn_ASID_SHIFT        48
> > >
> > > @@ -778,6 +779,9 @@ static void arm_smmu_init_context_bank(struct
> > > arm_smmu_domain *smmu_domain,
> > >              reg = pgtbl_cfg->arm_lpae_s1_cfg.tcr;
> > >              reg2 = pgtbl_cfg->arm_lpae_s1_cfg.tcr >> 32;
> > >              reg2 |= TTBCR2_SEP_UPSTREAM;
> > > +            if (smmu->model == CAVIUM_SMMUV2 &&
> >
> > I'd be inclined to say "smmu->version == ARM_SMMU_V2" there, rather than
> > make it Cavium-specific - we enable 16-bit VMID unconditionally where
> > supported, so I don't see any reason not to handle 16-bit ASIDs in the
> > same manner.
>
> I agree, I will enable 16-bit ASID for ARM_SMMU_V2.

Actually, the ARM_SMMU_CTX_FMT_AARCH64 context check is all we need here:

+			if (cfg->fmt == ARM_SMMU_CTX_FMT_AARCH64)
+				reg2 |= TTBCR2_AS;

Thanks,
Tomasz