Re: [PATCH v3 03/32] irqchip/gic-v3: Workaround Cavium TX1 erratum when reading GICD_TYPER2

From: Marc Zyngier
Date: Tue Mar 10 2020 - 07:41:13 EST

Hi Robert,

On 2020-03-09 22:11, Robert Richter wrote:
On 24.12.19 11:10:26, Marc Zyngier wrote:
Despite the architecture spec being extremely clear about the fact
that reserved registers in the GIC distributor memory map are RES0
(and thus are not allowed to generate an exception), the Cavium
ThunderX (aka TX1) SoC explodes as such:

[ 0.000000] GICv3: GIC: Using split EOI/Deactivate mode
[ 0.000000] GICv3: 128 SPIs implemented
[ 0.000000] GICv3: 0 Extended SPIs implemented
[ 0.000000] Internal error: synchronous external abort: 96000210 [#1] SMP
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.4.0-rc4-00035-g3cf6a3d5725f #7956
[ 0.000000] Hardware name: cavium,thunder-88xx (DT)
[ 0.000000] pstate: 60000085 (nZCv daIf -PAN -UAO)
[ 0.000000] pc : __raw_readl+0x0/0x8
[ 0.000000] lr : gic_init_bases+0x110/0x560
[ 0.000000] sp : ffff800011243d90
[ 0.000000] x29: ffff800011243d90 x28: 0000000000000000
[ 0.000000] x27: 0000000000000018 x26: 0000000000000002
[ 0.000000] x25: ffff8000116f0000 x24: ffff000fbe6a2c80
[ 0.000000] x23: 0000000000000000 x22: ffff010fdc322b68
[ 0.000000] x21: ffff800010a7a208 x20: 00000000009b0404
[ 0.000000] x19: ffff80001124dad0 x18: 0000000000000010
[ 0.000000] x17: 000000004d8d492b x16: 00000000f67eb9af
[ 0.000000] x15: ffffffffffffffff x14: ffff800011249908
[ 0.000000] x13: ffff800091243ae7 x12: ffff800011243af4
[ 0.000000] x11: ffff80001126e000 x10: ffff800011243a70
[ 0.000000] x9 : 00000000ffffffd0 x8 : ffff80001069c828
[ 0.000000] x7 : 0000000000000059 x6 : ffff8000113fb4d1
[ 0.000000] x5 : 0000000000000001 x4 : 0000000000000000
[ 0.000000] x3 : 0000000000000000 x2 : 0000000000000000
[ 0.000000] x1 : 0000000000000000 x0 : ffff8000116f000c
[ 0.000000] Call trace:
[ 0.000000] __raw_readl+0x0/0x8
[ 0.000000] gic_of_init+0x188/0x224
[ 0.000000] of_irq_init+0x200/0x3cc
[ 0.000000] irqchip_init+0x1c/0x40
[ 0.000000] init_IRQ+0x160/0x1d0
[ 0.000000] start_kernel+0x2ec/0x4b8
[ 0.000000] Code: a8c47bfd d65f03c0 d538d080 d65f03c0 (b9400000)

when reading the GICv4.1 GICD_TYPER2 register, which is unexpected...

Work around it by adding a new quirk flagging all the A1 revisions
of the distributor, but it remains unknown whether this could affect
other revisions of this SoC (or even other SoCs from the same silicon
vendor).

Signed-off-by: Marc Zyngier <maz@xxxxxxxxxx>
---
drivers/irqchip/irq-gic-v3.c | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 286f98222878..640d4db65b78 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -34,6 +34,7 @@
#define GICD_INT_NMI_PRI (GICD_INT_DEF_PRI & ~0x80)

#define FLAGS_WORKAROUND_GICR_WAKER_MSM8996 (1ULL << 0)
+#define FLAGS_WORKAROUND_GICD_TYPER2_TX1 (1ULL << 1)

struct redist_region {
void __iomem *redist_base;
@@ -1464,6 +1465,15 @@ static bool gic_enable_quirk_msm8996(void *data)
return true;
}

+static bool gic_enable_quirk_tx1(void *data)
+{
+ struct gic_chip_data *d = data;
+
+ d->flags |= FLAGS_WORKAROUND_GICD_TYPER2_TX1;
+
+ return true;
+}
+
static bool gic_enable_quirk_hip06_07(void *data)
{
struct gic_chip_data *d = data;
@@ -1502,6 +1512,12 @@ static const struct gic_quirk gic_quirks[] = {
.mask = 0xffffffff,
.init = gic_enable_quirk_hip06_07,
},
+ {
+ .desc = "GICv3: Cavium TX1 GICD_TYPER2 erratum",

There is no errata number yet.

Please let me know when/if you obtain one.


+ .iidr = 0xa100034c,
+ .mask = 0xfff00fff,
+ .init = gic_enable_quirk_tx1,

All TX1 and OcteonTX parts are affected, which is a0-a7 and b0-b7. So
the iidr/mask should be:

.iidr = 0xa000034c,
.mask = 0xe8f00fff,

Thanks, that's pretty helpful. I'll update the patch with these values
and the corresponding description.

+ },
{
}
};
@@ -1577,7 +1593,12 @@ static int __init gic_init_bases(void __iomem *dist_base,
pr_info("%d SPIs implemented\n", GIC_LINE_NR - 32);
pr_info("%d Extended SPIs implemented\n", GIC_ESPI_NR);

- gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base + GICD_TYPER2);
+ /*
+ * ThunderX1 explodes on reading GICD_TYPER2, in total violation
+ * of the spec (which says that reserved addresses are RES0).
+ */
+ if (!(gic_data.flags & FLAGS_WORKAROUND_GICD_TYPER2_TX1))
+ gic_data.rdists.gicd_typer2 = readl_relaxed(gic_data.dist_base + GICD_TYPER2);

You already said that checking for ArchRev of GICD_PIDR2 isn't an
option here. Though, it could...

Once GICv3.2 starts using this register as well (because GICD_TYPER is
already completely full), we'd have to fix it again. There is also the thing
you hinted at in the other thread: TX1 will generate a SEA on every reserved
GICD registers, so we may need to protect more than just this one over time,
and maybe more than just in the distributor.

Thanks,

M.
--
Jazz is not dead. It just smells funny...