[PATCH v3 0/2] mm: optimize HugeTLB bootmem hugepage init

[Li Zhe]

This series speeds up the fallback initialization path for HugeTLB tail
struct pages during boot-time hugepage allocation, when HugeTLB vmemmap
optimization (HVO) is unavailable or fails.

Patch 1 factors the PFN-dependent template refresh from the ZONE_DEVICE
template-copy path into memmap_init_refresh_pfn_fields().  The
ZONE_DEVICE path keeps its memcpy_nontemporal() copy primitive.

Patch 2 adds a HugeTLB-specific copy helper in
hugetlb_folio_init_tail_vmemmap(): initialize the first tail page
normally, then initialize the remaining tail pages by copying that page
with memcpy() and refreshing only PFN-dependent fields in the destination
page.

In the earlier discussion, Mike asked whether the same kind of
template-based initialization also helps the ordinary struct page
initialization path:

https://lore.kernel.org/all/akfB6Qe-jKDEMUxz@xxxxxxxxxx/

I tested that direction as well, but did not see a measurable benefit
for ordinary struct page initialization. The measurable improvement I
could find is in the HugeTLB bootmem hugepage tail path, where gigantic
pages can require initializing a large number of tail struct pages when
HVO does not cover the allocation.

Tested on a v7.4-rc2 base with 4K base pages and 64 boot-time 1G
hugepages, measured with a temporary jiffies-based printk.  HVO is
unavailable on arm64; on x86-64, hugetlb_free_vmemmap=off was used to
exercise the fallback path.

  arch    base      with this patch  reduction
  arm64   360.8 ms  256.0 ms         29.05%
  x86-64  171.0 ms  165.0 ms          3.51%

I also checked memcpy_nontemporal() for the HugeTLB helper, but it did
not show a similar optimization to the PMEM/ZONE_DEVICE template-copy
path.  One likely reason is that HugeTLB initializes tail struct pages in
per-hugepage chunks, while the ZONE_DEVICE path is closer to a long
streaming initialization over a device range.  In this HugeTLB fallback
path, the measured x86-64 result was better with memcpy(), so this series
keeps memcpy_nontemporal() in the ZONE_DEVICE helper and uses memcpy() in
the HugeTLB helper.

The temporary measurement instrumentation is not part of this series.

Li Zhe (2):
  mm/mm_init: factor out memmap_init_refresh_pfn_fields()
  mm/hugetlb: optimize bootmem hugepage tail vmemmap init

 mm/hugetlb.c | 23 ++++++++++++++++++-----
 mm/mm_init.c | 11 ++++++-----
 mm/mm_init.h | 11 +++++++++++
 3 files changed, 35 insertions(+), 10 deletions(-)

---
v2: https://lore.kernel.org/all/20261110114859.76343-1-lizhe.67@xxxxxxxxxxxxx/
v1: https://lore.kernel.org/all/20261105120615.16817-1-lizhe.67@xxxxxxxxxxxxx/#t

Changes in v3:
- Share only the PFN-dependent template refresh.  Keep
  memcpy_nontemporal() in the ZONE_DEVICE helper and use memcpy() in
  the HugeTLB helper.
- Copy HugeTLB tail pages from the first initialized tail page and
  refresh PFN-dependent fields in the destination, avoiding a local
  template in the HugeTLB helper.
- Restore the empty-range guard in hugetlb_folio_init_tail_vmemmap()
  after Sashiko reported a valid powerpc 8xx case.

Changes in v2:
- Add a comment for the template-before-copy ordering, as suggested by
  Muchun Song.
- Drop the empty-range guard in hugetlb_folio_init_tail_vmemmap()
  as suggested by Balbir Singh and Mike Rapoport.
-- 
2.20.1