Re: [PATCH] x86/mm: Revert INVLPGB optimization for set_memory code

[Hellstrom, Thomas]

On Tue, 2026-04-21 at 08:19 -0700, Dave Hansen wrote:
> 
> From: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
> 
> tl;dr: Revert an INVLPGB optimization that did not properly handle
> discontiguous virtual addresses.
> 
> Full story:
> 
> I got a report from some graphics (i915) folks that bisected a
> regression in their test suite to 86e6815b316e ("x86/mm: Change
> cpa_flush() to call flush_kernel_range() directly").  There was a bit
> of flip-flopping on the exact bisect, but the code here does seem
> wrong to me. The i915 folks were calling set_pages_array_wc(), so
> using the CPA_PAGES_ARRAY mode.
> 
> Basically, the 'struct cpa_data' can wrap up all kinds of page table
> changes.  Some of these are virtually contiguous, but some are very
> much not which is one reason why there are ->vaddr and ->pages
> arrays.
> 
> 86e6815b316e made the mistake of assuming that the virtual addresses
> in the cpa_data are always contiguous. It got things right when
> neither
> CPA_ARRAY/CPA_PAGES_ARRAY is used, but theoretically wrong when
> either
> of those is used.
> 
> In the i915 case, it probably failed to flush some WB TLB entries and
> install WC ones, leaving some data in the caches and not flushing it
> out to where the device could see it. That eventually caused graphics
> problems.
> 
> Revert the INVLPGB optimization. It can be reintroduced later, but it
> will need to be a bit careful about the array modes.
> 
> Signed-off-by: Dave Hansen <dave.hansen@xxxxxxxxxxxxxxx>
> Reported-by: Cui, Ling <ling.cui@xxxxxxxxx>
> Cc: Yu-cheng Yu <yu-cheng.yu@xxxxxxxxx>
> Cc: Rik van Riel <riel@xxxxxxxxxxx>
> Cc: Hellstrom, Thomas <thomas.hellstrom@xxxxxxxxx>
> Cc: Andy Lutomirski <luto@xxxxxxxxxx>
> Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
> Cc: Thomas Gleixner <tglx@xxxxxxxxxx>
> Cc: Ingo Molnar <mingo@xxxxxxxxxx>
> Cc: Borislav Petkov <bp@xxxxxxxxx>
> Cc: x86@xxxxxxxxxx

Reviewed-by: Thomas Hellström <thomas.hellstrom@xxxxxxxxxxxxxxx>

Thanks,
Thomas


> ---
> 
>  b/arch/x86/mm/pat/set_memory.c |   20 +++++++++++++-------
>  1 file changed, 13 insertions(+), 7 deletions(-)
> 
> diff -puN arch/x86/mm/pat/set_memory.c~revert-86e6815b316ec0ea8
> arch/x86/mm/pat/set_memory.c
> --- a/arch/x86/mm/pat/set_memory.c~revert-
> 86e6815b316ec0ea8	2026-04-21 06:56:29.208686362 -0700
> +++ b/arch/x86/mm/pat/set_memory.c	2026-04-21
> 06:57:54.946011984 -0700
> @@ -399,6 +399,15 @@ static void cpa_flush_all(unsigned long
>  	on_each_cpu(__cpa_flush_all, (void *) cache, 1);
>  }
>  
> +static void __cpa_flush_tlb(void *data)
> +{
> +	struct cpa_data *cpa = data;
> +	unsigned int i;
> +
> +	for (i = 0; i < cpa->numpages; i++)
> +		flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i)));
> +}
> +
>  static int collapse_large_pages(unsigned long addr, struct list_head
> *pgtables);
>  
>  static void cpa_collapse_large_pages(struct cpa_data *cpa)
> @@ -435,7 +444,6 @@ static void cpa_collapse_large_pages(str
>  
>  static void cpa_flush(struct cpa_data *cpa, int cache)
>  {
> -	unsigned long start, end;
>  	unsigned int i;
>  
>  	BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
> @@ -445,12 +453,10 @@ static void cpa_flush(struct cpa_data *c
>  		goto collapse_large_pages;
>  	}
>  
> -	start = fix_addr(__cpa_addr(cpa, 0));
> -	end =   start + cpa->numpages * PAGE_SIZE;
> -	if (cpa->force_flush_all)
> -		end = TLB_FLUSH_ALL;
> -
> -	flush_tlb_kernel_range(start, end);
> +	if (cpa->force_flush_all || cpa->numpages >
> tlb_single_page_flush_ceiling)
> +		flush_tlb_all();
> +	else
> +		on_each_cpu(__cpa_flush_tlb, cpa, 1);
>  
>  	if (!cache)
>  		goto collapse_large_pages;
> _