[PATCH v2 2/3] mm/madvise: skip device-private PMDs in cold and pageout walks

[Usama Arif]

madvise_cold_or_pageout_pte_range() takes pmd_trans_huge_lock(), whose
pmd_is_huge() check returns true for a device-private PMD. The subsequent
!pmd_present() branch has a VM_BUG_ON() asserting migration is the only
allowed non-present case; a device-private PMD trips it.

Allow device-private PMDs in that non-present assertion and continue to
huge_unlock before calling pmd_folio(). This keeps the assertion for
unexpected PMD softleafs while skipping device-private PMDs like other
non-present PMDs in this path.

Potential trigger: an HMM-based GPU driver races with
madvise(MADV_COLD)/MADV_PAGEOUT: pmd_trans_huge(*pmd) reads true, then
migrate_vma_pages() flips the PMD to a device-private entry before the
PMD lock is acquired.

Reported-by: sashiko-bot <sashiko-bot@xxxxxxxxxx>
Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=6
Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations")
Cc: <stable@xxxxxxxxxxxxxxx>
Reviewed-by: Joshua Hahn <joshua.hahnjy@xxxxxxxxx>
Reviewed-by: Zi Yan <ziy@xxxxxxxxxx>
Reviewed-by: Balbir Singh <balbirs@xxxxxxxxxx>
Signed-off-by: Usama Arif <usama.arif@xxxxxxxxx>
---
 mm/madvise.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/madvise.c b/mm/madvise.c
index 9292f60b19aa..1065b5a84ea7 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -389,7 +389,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 
 		if (unlikely(!pmd_present(orig_pmd))) {
 			VM_BUG_ON(thp_migration_supported() &&
-					!pmd_is_migration_entry(orig_pmd));
+					!pmd_is_migration_entry(orig_pmd) &&
+					!pmd_is_device_private_entry(orig_pmd));
 			goto huge_unlock;
 		}
 
-- 
2.53.0-Meta