[PATCH v3 0/5] x86/mm: Allow preemption while waiting for kernel TLB flushes

[Chuyi Zhou]

Changes in v3:
  - Patch 4: Retain struct flush_tlb_info in the kernel range helpers and
    initialize start/end directly, keeping the shared descriptor for
    future consolidation of the kernel and user paths (Nadav).

v2: https://lore.kernel.org/cover.1791179421.git.zhouchuyi@xxxxxxxxxxxxx/

Changes in v2:
  - Patch 2: Reuse flush_tlb_all() for kernel ranges promoted to a full
    flush and remove kernel_tlb_flush_all() (Sebastian).
  - Patch 4: Drop the explicit TLB_FLUSH_ALL check on the end argument.
    Callers pass actual address ranges and can use flush_tlb_all() for
    unconditional full flushes (Sebastian).

v1: https://lore.kernel.org/20260921095359.3784458-1-zhouchuyi@xxxxxxxxxxxxx/

This series follows up on the IPI completion preemption work [1] and
addresses the deferred flush_tlb_kernel_range() changes. The generic SMP
completion waits are already preemptible, and commit a5a162fe1ae1
("x86/mm: Re-enable preemption before flush_tlb_multi()") allows the mm
flush paths to use them. The kernel-range change was deferred during
review [2].

Container teardown and BPF map destruction can trigger kernel TLB
flushes through reclamation of unused per-CPU memory. Services that
spawn and reap many workers can also trigger flushes when vmalloc-backed
kernel stacks are reclaimed.

On the IPI backend, flush_tlb_kernel_range() flushes the local CPU,
sends flush requests to all other online CPUs, and waits synchronously
for completion. The target set is system-wide even when the container
or application is confined to a small subset of CPUs.

The synchronous wait can become longer as the number of online CPUs
grows. Completion depends on the slowest participating CPU, so a remote
CPU with interrupts disabled can delay the entire operation.
flush_tlb_kernel_range() keeps preemption disabled throughout that wait,
delaying higher-priority tasks on the initiating CPU.

The kernel path still uses init_flush_tlb_info(), which initializes
initiating_cpu with smp_processor_id(). Simply removing the outer
preemption guard would allow that initialization to run in a preemptible
context and could trigger a CONFIG_DEBUG_PREEMPT warning. An earlier
revision of the IPI preemption work used raw_smp_processor_id() to
suppress that warning, but this also removed the CPU-pinning check from
the shared initializer used by the mm paths.

The kernel callbacks consume only start and end. This series initializes
those fields directly in struct flush_tlb_info and selects full or range
flushes with the shared threshold predicate. This removes the unused
initiating_cpu initialization while retaining the shared descriptor for
future consolidation of the kernel and user TLB invalidation paths.
The smp_processor_id() check remains in the initializer for the mm paths.

Removing the outer preemption guard then lets higher-priority tasks
preempt the final IPI completion wait when the calling context permits
it. Both flush backends remain synchronous, and the INVLPGB helpers
keep their required preemption protection.

The changes are split into five patches:

  1. Account for kernel TLB flush requests in NR_TLB_REMOTE_FLUSH,
     including ranges promoted to full flushes and both IPI and INVLPGB
     backends.

  2. Remove kernel_tlb_flush_all() and use flush_tlb_all() for kernel
     ranges promoted to a full flush, sharing backend dispatch and
     accounting while counting each request once.

  3. Extract the range-to-full-flush threshold predicate without changing
     its arithmetic or the existing flush policy.

  4. Initialize start/end directly in struct flush_tlb_info for kernel
     range flushes and use the shared threshold predicate to select full
     or range flushes. Keep the outer preemption guard and the existing
     kernel helper interfaces.

  5. Remove the outer preemption guard from flush_tlb_kernel_range().
     Keep the INVLPGB range loop and TLBSYNC protected inside their
     backend helper; the full INVLPGB helper already has that protection.

[1] https://lore.kernel.org/lkml/20260709122933.4021501-1-zhouchuyi@xxxxxxxxxxxxx/
[2] https://lore.kernel.org/9cd743e8-4d60-4a5b-906f-07e4ae82dafb@xxxxxxxxxxxxx/

Chuyi Zhou (5):
  x86/mm: Account for remote kernel TLB flush requests
  x86/mm: Share the full TLB flush dispatch
  x86/mm: Extract the TLB range flush threshold check
  x86/mm: Initialize kernel TLB flush requests directly
  x86/mm: Re-enable preemption before waiting for kernel TLB flushes

 arch/x86/mm/tlb.c | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)


base-commit: e81ee06308379a5f2ededf997bcf17551bce5db7