Re: [PATCH] mm: synchronize with PTE table reclaim for all folio sizes
From: Baolin Wang
Date: Thu Nov 05 2026 - 21:54:27 EST
On 11/6/26 10:33 AM, Lance Yang wrote:
On Fri, Nov 06, 2026 at 09:57:47AM +0800, Baolin Wang wrote:
On 11/5/26 9:22 PM, Lance Yang wrote:
On Thu, Nov 05, 2026 at 11:06:29AM +0100, David Hildenbrand (Arm) wrote:
On 11/5/26 10:00, Baolin Wang wrote:
CC Barry, who added the TTU_SYNC in shrink_folio_list().
On 11/5/26 1:48 PM, Lance Yang wrote:
On 2026/11/5 12:54, Lance Yang wrote:
Good catch!
Right. Whether the folio can be PMD-mapped doesn't tell us whether its
PTE table can be reclaimed, so those extra checks can skip synchronization
that the caller explicitly asked for ;)
Ouch ... one thing I missed ... what about order-0 reclaim?
shrink_folio_list() only sets TTU_SYNC for large folio, so can't we still
hit the same race there?
Thanks Lance. But this matches the purpose of the TTU_SYNC flag, which is used
to avoid racy checks with PVMW_SYNC. If no TTU_SYNC flag is set, it means the
caller can tolerate some racy checks.
For this case, I think we should set TTU_SYNC unconditionally in
shrink_folio_list(). Barry, what do you think?
That makes sense. I hope this doesn't throw a big wrench into our reclaim
performance numbers.
Can someone benchmark this?
But maybe we should really follow-up with the folio-size based optimization in
sync_with_folio_pmd_zap(), where we only need a memory barrier (and sufficient
documentation why something that cannot be PMD-mapped can get away cheaper).
Dug into the history a bit :D
TTU_SYNC goes back to Hugh's THP splitting fix [1]. A racing zap could
clear the PTE/PMD before updating mapcount, so the walker could skip the
lock and return before the accounting caught up.
Barry later enabled it for large-folio reclaim [2], to avoid skipping
temporarily cleared PTEs and ending up with a partially unmapped folio.
He mentioned that small folios had the same race, but with just one PTE,
they couldn't be partially unmapped. That's where the large-folio check
in shrink_folio_list() came from.
Then PTE table reclaim [3] added another way to end up with an empty PMD,
even for a folio that can't be PMD-mapped. The truncation fix [4] moved
the rmap flush before the PMD clear and extended synchronization in
zap_pmd_range() to cover those folios too.
Looking back at the truncation discussion [5], you also pointed out that
page_vma_mapped_walk() needed checking, and suggested a possible
folio-size based optimization in the helper.
Thanks.
Cheers!
[1] https://lore.kernel.org/all/c1e95853-8bcd-d8fd-55fa-e7f2488e78f@xxxxxxxxxx/
[2] https://lore.kernel.org/all/20240306095219.71086-1-21cnbao@xxxxxxxxx/
[3] https://lore.kernel.org/all/92aba2b319a734913f18ba41e7d86a265f0b84e2.1733305182.git.zhengqi.arch@xxxxxxxxxxxxx/
[4] https://lore.kernel.org/all/14ebbd556cc4941f0ebc43da59fa6a2fdca25dc8.1793148609.git.baolin.wang@xxxxxxxxxxxxxxxxx/
[5] https://lore.kernel.org/all/08a0907f-8e3a-4071-a2e4-fd1411c446f3@xxxxxxxxxx/
So, how about starting with order-0 folios?
.......
atomic_add_negative() provides full ordering for mapcount updates. With
243cf1f67b36, rmap removal completes before clearing PMD, including when
another CPU reclaims an empty table under PTE lock. So smp_rmb() is enough
here, without an extra writer-side barrier.
I'm afraid the memory barrier here is subtle. __folio_remove_rmap() is
complex, and in the CONFIG_NO_PAGE_MAPCOUNT branch,
folio_sub_return_large_mapcount() doesn't provide full ordering, so we
may still need smp_wmb().
Yeah, agreed for large folios. In the diff above, only order-0 folios
use smp_rmb().
Note that order-0 folios take the fully ordered atomic_add_negative()
branch and skip the CONFIG_NO_PAGE_MAPCOUNT branch entirely.
Yes.
static __always_inline void __folio_remove_rmap(struct folio *folio,
struct page *page, int nr_pages, struct vm_area_struct *vma,
enum pgtable_level level)
{
...
switch (level) {
case PGTABLE_LEVEL_PTE:
if (!folio_test_large(folio)) {
nr = atomic_add_negative(-1, &folio->_mapcount);
break;
}
if (IS_ENABLED(CONFIG_NO_PAGE_MAPCOUNT)) {
nr = folio_sub_return_large_mapcount(folio, nr_pages, vma);
...
}
...
}
...
}
And large folios still take the PMD lock :D
IIUC, the memory barrier optimization targets all non-PMD-sized folios (including large folios). That is, for PMD-sized large folios we still use the original PMD lock, while for other non-PMD-sized folios we can use memory barriers to sync.