Re: arm64: tegra186: bpmp: kernel crash while decompressing initrd

[Robin Murphy]

On 2020-05-08 9:40 am, Mian Yousaf Kaukab wrote:
> I am seeing following kernel crash on Jetson TX2. Board is flashed with
> firmware bits from L4T R32.4.2 with upstream u-boot. Crash always
> happens while decompressing initrd. Initrd is approximately 80 MiB in
> size and compressed with xz (xz --check=crc32 --lzma2=dict=32MiB).
> Crash is not observed if the same initrd is compressed with gzip.
> [1] was a previous attempt to workaround the same issue.
>
> [    0.651168] Trying to unpack rootfs image as initramfs...
> [    2.890171] SError Interrupt on CPU0, code 0xbf40c000 -- SError
> [    2.890174] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G S                5.7.0-rc4-next-20200505 #22
> [    2.890175] Hardware name: nvidia p2771-0000/p2771-0000, BIOS 2020.04-rc3 03/25/2020
> [    2.890176] pstate: 20000005 (nzCv daif -PAN -UAO BTYPE=--)
> [    2.890177] pc : lzma_main+0x648/0x908
> [    2.890178] lr : lzma_main+0x330/0x908
> [    2.890179] sp : ffff80001003bb70
> [    2.890180] x29: ffff80001003bb70 x28: 0000000004d794a4
> [    2.890183] x27: 0000000004769941 x26: ffff0001eb064000
> [    2.890185] x25: ffff0001eb060028 x24: 0000000000000002
> [    2.890187] x23: 0000000000000003 x22: 0000000000000007
> [    2.890189] x21: 0000000000611f4b x20: ffff0001eb060000
> [    2.890192] x19: ffff80001003bcb8 x18: 0000000000000068
> [    2.890194] x17: 00000000000000c0 x16: fffffe00076b2108
> [    2.890196] x15: 0000000000000800 x14: 0000000000ffffff
> [    2.890198] x13: 0000000000000001 x12: ffff0001eb060000
> [    2.890200] x11: 0000000000000600 x10: ffff0001eb060028
> [    2.890202] x9 : 00000000ffbb2a08 x8 : 0000000000000ed0
> [    2.890204] x7 : 00000000011553ec x6 : 0000000000000000
> [    2.890206] x5 : 0000000000000000 x4 : 0000000000000006
> [    2.890208] x3 : 00000000015a29e4 x2 : ffff0001eb062d0c
> [    2.890210] x1 : 000000000000000c x0 : 000000000263de44
>
> With some debugging aid ported from Nvidia downstream kernel [2] the
> actual cause was found:
>
> [    0.761525] Trying to unpack rootfs image as initramfs...
> [    2.955499] CPU0: SError: mpidr=0x80000100, esr=0xbf40c000
> [    2.955502] CPU1: SError: mpidr=0x80000000, esr=0xbe000000
> [    2.955505] CPU2: SError: mpidr=0x80000001, esr=0xbe000000
> [    2.955506] CPU3: SError: mpidr=0x80000101, esr=0xbf40c000
> [    2.955507] ROC:CCE Machine Check Error:
> [    2.955508] ROC:CCE Registers:
> [    2.955509]  STAT: 0xb400000000400415
> [    2.955510]  ADDR: 0x400c00e7a00c
> [    2.955511]  MSC1: 0x80ffc
> [    2.955512]  MSC2: 0x3900000000800
> [    2.955513] --------------------------------------
> [    2.955514] Decoded ROC:CCE Machine Check:
> [    2.955515]  Uncorrected (this is fatal)
> [    2.955516]  Error reporting enabled when error arrived
> [    2.955517]  Error Code = 0x415
> [    2.955518]  Poison Error
> [    2.955518]  Command = NCRd (0xc)
> [    2.955519]  Address Type = Non-Secure DRAM
> [    2.955521]  Address = 0x30039e80 -- 30000000.sysram + 0x39e80
> [    2.955521]  TLimit = 0x3ff
> [    2.955522]  Poison Error Mask = 0x80
> [    2.955523]  More Info = 0x800
> [    2.955524]  Timeout Info = 0x0
> [    2.955525]          Poison Info = 0x800
> [    2.955526]          Read Request failed GSC checks
> [    2.955527]  Source = L2_1 (A57) (0x1)
> [    2.955528]  TID = 0xe
>
> IIUC, there was read request for 0x30039e80 from EL1/2 which failed.
> This address falls in the sysram security aperture and hence a read
> from normal mode failed.
>
> sysram is mapped at 0x3000_0000 to 0x3004_ffff and is managed by the
> sram driver (drivers/misc/sram.c). There are two reserved pools for
> BPMP driver communication at 0x3004_e000 and 0x3004_f000 of 0x1000
> bytes each.
>
> sram driver maps complete 0x3000_0000 to 0x3004_ffff range as normal
> memory.

That's your problem. It's not really worth attempting to reason about, 
the architecture says that anything mapped as Normal memory may be 
speculatively accessed at any time, so no amount of second-guessing is 
going to save you in general. Don't make stuff accessible to the kernel 
that it doesn't need to access, and especially don't make stuff 
accessible to the kernel if accessing it will kill the system.

> However, only the BPMP reserved pools (0x3004_e000 - 0x3004_ffff)
> are accessible from the kernel. Address 0x3003_9e80 is inaccessible
> from the kernel and a read to it (which I believe is speculative)
> causes the SError. Only driver which uses sysram is not initialized at
> this point (rootfs_initcall level). As since
> commit d70f5e541ab3 ("firmware: tegra: Make BPMP a regular driver")
> bpmp driver is initialized at device_initcall level.
>
> If none of the drivers on the kernel side using 0x3003_9e80 address
> range. Why a read to it occurs even speculatively? Could it be that
> some EL3 software didnât cleanup after itself properly? Any
> suggestions on debugging this issue further?
>
> Another solution suggested in [1] was to add no-memory-wc in sysram
> node in device-tree so that sysram is mapped as device-memory. Thus
> preventing any speculative access. However, it causes another set of
> issues with the bpmp driver. That's may be a discussion for another
> time.

AFAICS the truly correct solution is what Stephen initially suggested 
there - for the boot process to somehow describe which parts of SRAM 
have been reserved by Secure software and/or which parts remain 
Non-Secure, and for the kernel driver to only map and use the latter.

Robin.