Re: [PATCH v14 03/13] x86/mm: add INVLPGB support code

[Borislav Petkov]

On Tue, Mar 04, 2025 at 08:57:30AM -0800, Dave Hansen wrote:
> Why would __invlpg_all() need an 'addr' or 'nr_pages'? Shouldn't those be 0?

Yap, good idea. It makes the _all helper even better:

static inline void __invlpgb_all(unsigned long asid, unsigned long pcid, u8 flags)
{
        __invlpgb(asid, pcid, 0, 1, 0, flags);
}

> It's _better_ of course when it happens at a single site and it's close
> to a prototype for __invlpgb(). But it's still a magic '0' that it's
> impossible to make sense of without looking at the prototype.

Yes.

> Looking at the APM again... there really are three possible values for
> ECX[31]:
> 
>  0: increment by 4k
>  1: increment by 2M
>  X: Don't care, no increment is going to happen
> 
> What you wrote above could actually be written:
> 
> 	__invlpgb(asid, pcid, addr, nr_pages, 1, flags);
> 
> so the 0/1 is _actually_ completely random and arbitrary as far as the
> spec goes.

Yes.

> Why does it matter?
> 
> It enables you to do sanity checking. For example, we could actually
> enforce a rule that "no stride" can't be paired with any of the
> per-address invalidation characteristics:
> 
> 	if (stride == NO_STRIDE) {
> 		WARN_ON(flags & INVLPGB_FLAG_VA);
> 		WARN_ON(addr);
> 		WARN_ON(nr_pages);
> 	}
> 
> That's impossible if you pass a 'bool' in.
> 
> But, honestly, I'm deep into nitpick mode here. I think differentiating
> the three cases is worth it, but it's also not the hill I'm going to die
> on. ;)

Yap, and now I've massaged it so much so that it doesn't really need that
checking. Because I have exactly two calls which use the stride:

1.

static inline void __invlpgb_flush_user_nr_nosync(unsigned long pcid,
                                                  unsigned long addr,
                                                  u16 nr, bool stride)
{
        enum addr_stride str = stride ? PMD_STRIDE : PTE_STRIDE;
        u8 flags = INVLPGB_FLAG_PCID | INVLPGB_FLAG_VA;

        __invlpgb(0, pcid, addr, nr, str, flags);
}

This one is fine - I verify it.

2.

/* Flush addr, including globals, for all PCIDs. */
static inline void __invlpgb_flush_addr_nosync(unsigned long addr, u16 nr)
{
	__invlpgb(0, 0, addr, nr, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL);
}

This one controls it already.

So the only case where something could go bad is when one would use
__invlpgb() directly and that should hopefully be caught early enough.

But if you really want, I could add sanitization to __invlpgb() to massage it
into the right stride. And print a single warning - the big fat WARN* in an
inline functions are probably too much. Hm, I dunno...

Current diff ontop:

diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h
index e8561a846754..8ab21487d6ee 100644
--- a/arch/x86/include/asm/tlb.h
+++ b/arch/x86/include/asm/tlb.h
@@ -66,6 +66,11 @@ static inline void __invlpgb(unsigned long asid, unsigned long pcid,
 	asm volatile(".byte 0x0f, 0x01, 0xfe" :: "a" (rax), "c" (ecx), "d" (edx));
 }
 
+static inline void __invlpgb_all(unsigned long asid, unsigned long pcid, u8 flags)
+{
+	__invlpgb(asid, pcid, 0, 1, 0, flags);
+}
+
 static inline void __tlbsync(void)
 {
 	/*
@@ -84,6 +89,7 @@ static inline void __tlbsync(void)
 static inline void __invlpgb(unsigned long asid, unsigned long pcid,
 			     unsigned long addr, u16 nr_pages,
 			     enum addr_stride s, u8 flags) { }
+static inline void __invlpgb_all(unsigned long asid, unsigned long pcid, u8 flags) { }
 static inline void __tlbsync(void) { }
 #endif
 
@@ -121,7 +127,7 @@ static inline void __invlpgb_flush_user_nr_nosync(unsigned long pcid,
 /* Flush all mappings for a given PCID, not including globals. */
 static inline void __invlpgb_flush_single_pcid_nosync(unsigned long pcid)
 {
-	__invlpgb(0, pcid, 0, 1, PTE_STRIDE, INVLPGB_FLAG_PCID);
+	__invlpgb_all(0, pcid, INVLPGB_FLAG_PCID);
 }
 
 /* Flush all mappings, including globals, for all PCIDs. */
@@ -134,7 +140,7 @@ static inline void invlpgb_flush_all(void)
 	 * as it is cheaper.
 	 */
 	guard(preempt)();
-	__invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_FLAG_INCLUDE_GLOBAL);
+	__invlpgb_all(0, 0, INVLPGB_FLAG_INCLUDE_GLOBAL);
 	__tlbsync();
 }
 
@@ -148,7 +154,7 @@ static inline void __invlpgb_flush_addr_nosync(unsigned long addr, u16 nr)
 static inline void invlpgb_flush_all_nonglobals(void)
 {
 	guard(preempt)();
-	__invlpgb(0, 0, 0, 1, PTE_STRIDE, INVLPGB_MODE_ALL_NONGLOBALS);
+	__invlpgb_all(0, 0, INVLPGB_MODE_ALL_NONGLOBALS);
 	__tlbsync();
 }
 #endif /* _ASM_X86_TLB_H */

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette