Re: [PATCH] x86/mm: Fix boot with some memory above MAXMEM

[Kirill A. Shutemov]

On Tue, May 26, 2020 at 07:27:15AM -0700, Dave Hansen wrote:
> On 5/25/20 8:08 AM, Kirill A. Shutemov wrote:
> >>>> +	if (not_addressable) {
> >>>> +		pr_err("%lldGB of physical memory is not addressable in the paging mode\n",
> >>>> +		       not_addressable >> 30);
> >>>> +		if (!pgtable_l5_enabled())
> >>>> +			pr_err("Consider enabling 5-level paging\n");
> >> Could this happen at all when l5 is enabled?
> >> Does it mean we need kmap() for 64-bit?
> > It's future-profing. Who knows what paging modes we would have in the
> > future.
> 
> Future-proofing and firmware-proofing. :)
> 
> In any case, are we *really* limited to 52 bits of physical memory with
> 5-level paging?

Yes. It's architectural. SDM says "MAXPHYADDR is at most 52" (Vol 3A,
4.1.4).

I guess it can be extended with an opt-in feature and relevant changes to
page table structure. But as of today there's no such thing.

> Previously, we said we were limited to 46 bits, and now
> we're saying that the limit is 52 with 5-level paging:
> 
> #define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)
> 
> The 46 was fine with the 48 bits of address space on 4-level paging
> systems since we need 1/2 of the address space for userspace, 1/4 for
> the direct map and 1/4 for the vmalloc-and-friends area.  At 46 bits of
> address space, we fill up the direct map.
> 
> The hardware designers know this and never enumerated a MAXPHYADDR from
> CPUID which was higher than what we could cover with 46 bits.  It was
> nice and convenient that these two separate things matched:
> 1. The amount of physical address space addressable in a direct map
>    consuming 1/4 of the virtual address space.
> 2. The CPU-enumerated MAXPHYADDR which among other things dictates how
>    much physical address space is addressable in a PTE.
> 
> But, with 5-level paging, things are a little different.  The limit in
> addressable memory because of running out of the direct map actually
> happens at 55 bits (57-2=55, analogous to the 4-level 48-2=46).
> 
> So shouldn't it technically be this:
> 
> #define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 55 : 46)
> 
> ?

Bits above 52 are ignored in the page table entries and accessible to
software. Some of them got claimed by HW features (XD-bit, protection
keys), but such features require explicit opt-in on software side.

Kernel could claim bits 53-55 for the physical address, but it doesn't get
us anything: if future HW would provide such feature it would require
opt-in. On other hand claiming them now means we cannot use them for other
purposes as SW bit. I don't see a point.

-- 
 Kirill A. Shutemov