[RFC PATCH 00/13] arm64: Preemptible this_cpu_*() operations

[Mark Rutland]

Currently arm64's this_cpu_*() ops transiently disable preemption in
order to guarantee that the address generation and memory access(es)
occur on the same CPU, which has some unfortunate overhead. In
particular, re-enabling preemption requires a conditional call to
preempt_schedule[_notrace](), which has a number of adverse impacts on
code generation.

This series reworks arm64's this_cpu_*() operations such that they do
not need to disable preemption, avoiding the related overhead in the
fast paths. Instead, the ope register a "PCPU GPR" critical section,
during which which the entry code will apply a fixup to GPRs containing
the percpu offset and the generated percpu address.

The scheme is described in detail in patch 6, and is similar to the
approach Heiko Carstens applied to s390 [1].

For example, an outlined this_cpu_read() goes from:

| <outline_this_cpu_write_u64>:
|        paciasp
|        stp     x29, x30, [sp, #-16]!
|        mrs     x2, sp_el0
|        mov     x29, sp
|        ldr     w3, [x2, #8]
|        add     w3, w3, #0x1
|        str     w3, [x2, #8]
|        mrs     x3, tpidr_el1
|        str     x1, [x0, x3]
|        ldr     x0, [x2, #8]
|        sub     x0, x0, #0x1
|        str     w0, [x2, #8]
|        cbz     x0, 1f
|        ldr     x0, [x2, #8]
|        cbnz    x0, 2f
| 1:     bl      preempt_schedule_notrace
| 2:     ldp     x29, x30, [sp], #16
|        autiasp
|        ret

... to:

| <outline_this_cpu_write_u64>:
|        mrs     x2, sp_el0
|        mov     w3, #0x7c60
|        strh    w3, [x2, #20]
|        mrs     x3, tpidr_el1
|        str     x1, [x0, x3]
|        strh    wzr, [x2, #20]
|        ret

Comparing defconfig before and after, the Image size is reduced by 64K, though
.text is only reduced by ~7.5K:

| [mark@lakrids:~/src/linux]% ls -al Image-defconfig-*
| -rw-r--r-- 1 mark mark 41552384 Jul 28 11:07 Image-defconfig-after
| -rw-r--r-- 1 mark mark 41617920 Jul 28 11:01 Image-defconfig-v7.2-rc4
| [mark@lakrids:~/src/linux]% ls -al vmlinux-defconfig-*
| -rwxr-xr-x 1 mark mark 158239216 Jul 28 11:07 vmlinux-defconfig-after
| -rwxr-xr-x 1 mark mark 158699800 Jul 28 11:01 vmlinux-defconfig-v7.2-rc4
| [mark@lakrids:~/src/linux]% size vmlinux-defconfig-*
|    text    data     bss     dec     hex filename
| 20323522        21028511         730840 42082873        2822239 vmlinux-defconfig-after
| 20331206        21028447         730840 42090493        2823ffd vmlinux-defconfig-v7.2-rc4

>From looking at the generated code, there are some reasonable savings we
could make with a few changes to alternatives, notably:

* Patching the read of TPIDR_ELx using a callback. This would remove the
  need for replacement instructions, saving ~12K for defconfig, and
  meaning real instructions in .text would be packed more densely.
  There's work underway to apply similar patching for
  ARM64_WORKAROUND_DEVICE_LOAD_ACQUIRE.

* Allowing {relocation,veneer}-free replacement sequences to be placed
  in .altinst_rreplacement again. This would allow real instructions in
  .text to be packed more densely, with ~80K of .text moving into
  .altinstr_replacement where it would be freed after init.

  I've thrown togther a prototype [2].

I'm sending this as an RFC as:

(1) I haven't had the chance to benchmark this. While I think the
    changes look good from inspection, we should obviously get some
    actual performance figures.

(2) I haven't wired up everything for SDEI exception entry/return.
    This should only require some mechinical rework to the
    __sdei_asm_handler() assembly, and I've noted this as a TODO on
    patch 6.

    Patches 1 and 2 are complete as-is, and can be taken without the
    rest of the series.

(3) I'm not sure whether this_cpu_read() should use a plain LDR, or use
    LDA[P]R when LTO is in use to match READ_ONCE(). Using LDA[P]R 
    might require some refactoring of the existing READ_ONCE() code to
    share the logic.

    I've noted this as a question on patch 7.

Thanks,
Mark.

[1] https://lore.kernel.org/linux-s390/20260526055702.1429061-1-hca@xxxxxxxxxxxxx/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/alternatives/cleanup
 
Mark Rutland (13):
  arm64: preempt: Simplify and optimize __preempt_count_dec_and_test()
  arm64: preempt: Treat should_resched() as unlikely
  arm64: ptrace: Always inline pt_regs_[read,write}_reg()
  arm64: percpu: Factor out percpu offset asm
  arm64: gpr-num: Add wxN aliases for wN registers
  arm64: percpu: Add infrastructure for preemptible this_cpu_*() ops
  arm64: percpu: Implement preemptible read/write ops
  arm64: percpu: Implement preemptible void RMW ops
  arm64: percpu: Implement preemptible return RMW ops
  arm64: percpu: Implement preemptible XCHG ops
  arm64: percpu: Implement preemptible CMPXCHG ops
  arm64: percpu: Implement preemptible CMPXCHG128 ops
  arm64: percpu: Remove _pcp_protect*() wrappers

 arch/arm64/include/asm/gpr-num.h     |   5 +
 arch/arm64/include/asm/percpu.h      | 437 +++++++++++++++++++++------
 arch/arm64/include/asm/preempt.h     |  29 +-
 arch/arm64/include/asm/ptrace.h      |  12 +-
 arch/arm64/include/asm/thread_info.h |   1 +
 arch/arm64/kernel/asm-offsets.c      |   2 +
 arch/arm64/kernel/entry-common.c     |  38 +++
 arch/arm64/kernel/entry.S            |  13 +
 8 files changed, 417 insertions(+), 120 deletions(-)

-- 
2.30.2