Re: [PATCH] arm64: use x22 to save boot exception level

[Mark Rutland]

On Fri, Aug 30, 2019 at 03:23:53PM -0400, Andrew F. Davis wrote:
> On 8/29/19 5:47 AM, Mark Rutland wrote:
> > On Wed, Aug 28, 2019 at 01:33:18PM -0400, Andrew F. Davis wrote:

> We are seeing is a write-back from L3 cache. Our bootloader writes the
> kernel image with caches on, then after turning off caching but before
> handing off to Linux it clean/invalidates all cache lines by set/way.
> This cleans out the L1/L2 but leaves dirty lines in L3. Our platform
> doesn't really have a good way to clean L3 as it only provides cache
> maintenance operations by VA, not by line, so we would need to clean
> every VA address manually..

Ensuring that the Image is clean to the PoC is required by the arm64
boot protocol, which states that maintenance by VA may be necessary in
the presence of a system cache. See:

https://www.kernel.org/doc/html/latest/arm64/booting.html

... which states:

| The MMU must be off. Instruction cache may be on or off. The address
| range corresponding to the loaded kernel image must be cleaned to the
| PoC. In the presence of a system cache or other coherent masters with
| caches enabled, this will typically require cache maintenance by VA
| rather than set/way operations. 

Please fix your bootloader to meet this requirement. The kernel is not
in a position to fix this up, e.g. as while the MMU is off instruction
fetches could fetch stale data from the PoC.

You only need to clean the kernel Image to the PoC, rather than all of
memory, so you should be able to do that with a loop of DC CVAC
instructions covering the VA range of the kernel Image.

> Also want to point out, although this isn't a problem for most platforms
> what this code does here, with writing to a location as non-cacheable,
> is not architecturally safe as the running cores that do the reads have
> this section marked as cacheable when they read, therefor you have
> mismatched attributes. When this happens like this according to the ARM
> ARM we should do a cache invalidate after the write *and* before the
> 
> I would like to work this fix from the U-Boot side also, but in parallel
> I would like to reduce the mismatched attributes as much as possible on
> the kernel side like done here. So yes, we still will have issue with
> __early_cpu_boot_status, but that only seems to be needed in the failure
> to boot case, I'd like to fix that up as well at some later point.

If you haven't cleaned the Image to the PoC, there's no guarantee that
any portion of it can be safely executed with the MMU off, so I don't
think that makes sense -- please fix your bootloader first.

I am aware that there are potential problems with mismatched attributes,
the primary issue here being unexpected-data-cache-hit. However, were
that to occur no amount of cache maintenance can save us in the presence
of a live cacheable alias. Practically speaking that's mainly a problem
for virtual environments.

Thanks,
Mark.