RE: [PATCH] x86/hyperv: Account for lazy mm_cpumask maintenance

[Michael Kelley]

From: Naman Jain <namjain@xxxxxxxxxxxxxxxxxxx> Sent: Thursday, October 29, 2026 9:01 PM
> 
> On 10/30/2026 8:09 AM, Michael Kelley wrote:
> > From: Naman Jain <namjain@xxxxxxxxxxxxxxxxxxx> Sent: Wednesday, October 28, 2026 11:43 PM
> >>
> >> CPUs no longer running an mm may remain in mm_cpumask(). Hyper-V treats
> >> these CPUs as flush targets and bypasses the native callbacks that trim
> >> the mask. A single-threaded process can therefore keep issuing
> >> all-processor flushes after visiting all CPUs and returning to one CPU.
> >>
> >> Share the native target predicate and filter the mask before building
> >> either hypercall format or selecting HV_FLUSH_ALL_PROCESSORS. Preserve
> >> lazy-TLB exclusion and unfiltered page-table-freeing flushes, and use
> >> native callbacks for periodic trimming and ASID transitions.
> >>
> >> When no remote targets remain, request immediate trimming through native
> >> callbacks so stale memberships do not unnecessarily prevent later
> >> invalidations from using the local fast path. Leave lazy CPU-mask
> >> maintenance during context switches unchanged.
> >>
> >> Fixes: 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily")
> >> Signed-off-by: Naman Jain <namjain@xxxxxxxxxxxxxxxxxxx>
> >
> > I've read the commit message but haven't yet delved into the code for
> > this patch. But the commit message feels like it is missing some context,
> > particularly with the "Fixes:" tag.
> >
> > I recall the last significant change to hyperv_flush_tlb_multi(), which takes
> > CPUs in lazy tlb mode into account, and thereby avoids unnecessary flushing
> > (commit 493cc07385cf, made in March 2023). It was a case where the
> > native x86 flush code had gotten smarter, and the Hyper-V PV flush
> > functions had not been updated to be smarter in the same way. Against
> > that context, what is the situation here? Commit 209954cbc7d0 came a
> > year and a half later in November 2024. Did it make things worse for the
> > Hyper-V PV flush functions, and hence is deemed to have "broken"
> > them? Or is it another native flush improvement that the Hyper-V PV
> > flush functions just don't take advantage of?
> >
> > I'm also curious how the issue was noticed. Was there a real-world
> > performance problem, or is it a case of looking at the code and
> > noticing a native improvement that hadn't been applied to the
> > Hyper-V PV flush functions? The scenario of a single-threaded process
> > visiting all CPUs and then returning to one CPU seems contrived,
> > particularly as CPU counts get large. It may illustrate the improvement
> > that is possible, but to me seems unlikely to be how the problem was
> > originally noticed.
> >
> > I intended to review the code carefully, but the additional context
> > would be helpful in doing that review.
> >
> > Michael
> >
> 
> Thanks Michael for looking into this.
> 
> We first observed a performance regression between Linux 6.8 and 7.0
> on Azure MI300X VMs running SuperBench's eight-GPU PyTorch LSTM training
> benchmark, using distributed data parallelism in FP32 and FP16.
> Recreating DataLoader workers each epoch triggered long host-to-GPU
> copy stalls and timeouts. We bisected the regression to commit being
> fixed and then developed the reduced reproducer to isolate the excessive
> Hyper-V TLB flushes. The CPU-visiting test came out of that
> investigation, it was not how the problem was originally noticed.
> 
> 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") changed where
> stale mm_cpumask entries are removed. Previously, switching a CPU from
> mm A to mm B cleared its bit in A's mask. After that change, the bit is
> left set for removal by the native TLB flush callback when it finds a
> different mm loaded.
> 
> The Hyper-V hypercall path bypasses that callback, so historical
> memberships can persist. The lazy-TLB check from 493cc07385cf does not
> cover this case: a CPU actively running B can have is_lazy == false
> while still appearing in A's mask. Hyper-V then unnecessarily targets
> it when flushing A. If the historical memberships cover all present
> CPUs, the all-processors shortcut is selected before the per-CPU
> lazy-TLB checks.
> 
> So the later change did make the Hyper-V path worse for affected
> workloads, rather than merely adding a native optimization that
> Hyper-V could also benefit from. That is the reason for the Fixes tag.
> This is a performance regression, not a stale-translation correctness
> failure.

Thanks. That's exactly what I wanted to know as background
information. It will help a lot when reviewing your patch.

> 
> The reporter has also tested the patch on a 96-vCPU MI300X VM. The
> unpatched SuperBench workload hit the 600-second timeout; two patched
> runs completed in 424 and 436 seconds, with all eight ranks validated.
> The kernel configurations matched apart from kernel identity, and
> the GPU driver source and executable text were unchanged.

Good info. That's a pretty significant difference in a macro-level
benchmarking tool.

Michael

> 
> For the reduced reproducer, the 96-CPU-history case dropped from
> 109.196 us per iteration to 1.594 and 1.616 us in the two patched runs,
> about a 98.5% reduction. Each iteration contains two mprotect() calls
> and a write, these are not individual hypercall timings.
> 
> The patch preserves your lazy-TLB exclusion and lazy CPU-mask
> maintenance during context switches. It filters the targets before
> selecting the hypercall format or the all-processors shortcut, and
> uses native callbacks for mask trimming.
> 
> I restricted the commit msg to the mentioned 32 vCPU VM because that was
> the one I got my hands on. I'll try to update the commit message or the
> comments to explain the real workload, bisection, and distinction
> between lazy TLB mode and lazy CPU-mask maintenance. But I'll wait for
> any review comments to address them together.
> 
> Thanks,
> Naman
> 
> >> ---
> >> On a 32-vCPU Hyper-V guest, after a single-threaded process visited all
> >> CPUs and returned to one CPU, the reproducer's median time per iteration
> >> (two mprotect() calls and a write) dropped from 17.235 us to 1.622 us,
> >> a 90.6% reduction. This could be more on higher vCPU VMs.
> >>
> <snip>
>