Re: [PATCH] mm: shmem: limit forced anonymous shmem huge orders to MAX_PAGECACHE_ORDER

From: Luiz Capitulino (Red Hat)

Date: Tue Nov 17 2026 - 21:40:21 EST




On 11/17/26 7:16 AM, Jingyi Sun wrote:
When the top-level shmem_enabled knob is set to "force",
shmem_allowable_huge_orders() returns huge_shmem_orders_inherit for
anonymous shmem without masking it with THP_ORDERS_ALL_FILE_DEFAULT,
unlike the other return path at the end of the same function.

huge_shmem_orders_inherit defaults to BIT(HPAGE_PMD_ORDER). On arm64
with 64K base pages HPAGE_PMD_ORDER is 13, while MAX_PAGECACHE_ORDER
is 11 because of the xarray split limit. With shmem_enabled=force, an
anonymous MAP_SHARED mapping can therefore get a 512M (order-13) folio
in the page cache. Splitting it later, e.g. on swap-out from
shmem_writeout(), triggers

WARNING: lib/xarray.c:1051 at xas_split_alloc

and the split fails, so the folio can never be swapped out.

This is the same problem that commit 9fd154ba926b ("mm/shmem: disable
PMD-sized page cache if needed") and commit b66b1b71d7ff ("mm: shmem:
avoid allocating huge pages larger than MAX_PAGECACHE_ORDER for shmem")
fixed for the other paths; the force branch was left out. Mask it with
THP_ORDERS_ALL_FILE_DEFAULT as well.

Reproduced on v7.3-rc2 in QEMU (arm64, 64K pages, -cpu cortex-a72,
4G RAM, zram swap): with shmem_enabled=force, a PMD-aligned
MAP_SHARED|MAP_ANONYMOUS mapping of one PMD size got a single
PMD-mapped folio (ShmemPmdMapped: 524288 kB) and MADV_PAGEOUT
triggered the warning above in 3/3 runs. With this change: 0/3.
shmem_enabled=always and 4K pages did not show the warning. The code is
unchanged in v7.4-rc3 and in mm-unstable, but the test has not been
re-run on those trees.

The issue was found with a research static analyzer for
cross-architecture dependency bugs and confirmed with the QEMU
reproducer described above.

Fixes: e7a2ab7b3bb5 ("mm: shmem: add mTHP support for anonymous shmem")
Cc: stable@xxxxxxxxxxxxxxx
Cc: Luiz Capitulino <luizcap@xxxxxxxxxx>
Assisted-by: LLM
Signed-off-by: Jingyi Sun <1045704958@xxxxxx>
---
Note: this touches the same line as "mm: thp: always enable mTHP
support" (Luiz Capitulino, v8 14/14), which is still under review. That
change masks the force branch with ~disabled_orders but does not add
THP_ORDERS_ALL_FILE_DEFAULT either, so it does not address this issue.
Happy to rebase on top of that series if it lands first.

I'm late with v9 so no way my series will land before this patch, but
thanks for the consideration :)


Link: https://lore.kernel.org/all/752f528f0fed5cdc9de12b54260b0495d4e5a6cb.1789695931.git.luizcap@xxxxxxxxxx/

mm/shmem.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index 10cef89b00b0..41bb3f2c6b63 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2114,7 +2114,8 @@ unsigned long shmem_allowable_huge_orders(struct inode *inode,
* means non-PMD sized THP can not override 'huge' mount option now.
*/
if (shmem_huge == SHMEM_HUGE_FORCE)
- return READ_ONCE(huge_shmem_orders_inherit);
+ return THP_ORDERS_ALL_FILE_DEFAULT &
+ READ_ONCE(huge_shmem_orders_inherit);
/* Allow mTHP that will be fully within i_size. */
mask |= shmem_get_orders_within_size(inode, within_size_orders, index, 0);

base-commit: 666539eecdf4db623f5e3c3e55cf9f2dfad24aa4