[PATCH v3 2/2] mm/shmem: restore SHMEM_HUGE_DENY precedence over MADV_COLLAPSE

[Zach O'Keefe]

SHMEM_HUGE_DENY is for emergency use by the admin, to disable allocation
of shmem huge pages if, for example, a dangerous bug is found in their
usage: see "deny" in Documentation/mm/transhuge.rst.  An app using
madvise(,,MADV_COLLAPSE) should not be allowed to override it: restore
its precedence over shmem_huge_force.

Restore SHMEM_HUGE_DENY precedence over MADV_COLLAPSE.

Fixes: 7c6c6cc4d3a2 ("mm/shmem: add flag to enforce shmem THP in hugepage_vma_check()")
Suggested-by: Hugh Dickins <hughd@xxxxxxxxxx>
Signed-off-by: Zach O'Keefe <zokeefe@xxxxxxxxxx>
Cc: Yang Shi <shy828301@xxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx
---
v2->v3: Add 'Cc: stable@xxxxxxxxxxxxxxx' as per stable-kernel-rules.
v1->v2: Update changelog, and add note explaining rationale for
	backporting  (Andrew Morton).

Request to backport this to 6.1.X stable.  We'd like SHMEM_HUGE_DENY to
take precedence over MADV_COLLAPSE.  If we make this change later, it
will be a userspace API change.  As such, 6.1 cannot be allowed to
continue as-is, and we should fix up the code there.
---
 mm/shmem.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mm/shmem.c b/mm/shmem.c
index c301487be5fb..0005ab2c29af 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -478,12 +478,10 @@ bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
 	if (vma && ((vma->vm_flags & VM_NOHUGEPAGE) ||
 	    test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags)))
 		return false;
-	if (shmem_huge_force)
-		return true;
-	if (shmem_huge == SHMEM_HUGE_FORCE)
-		return true;
 	if (shmem_huge == SHMEM_HUGE_DENY)
 		return false;
+	if (shmem_huge_force || shmem_huge == SHMEM_HUGE_FORCE)
+		return true;
 
 	switch (SHMEM_SB(inode->i_sb)->huge) {
 	case SHMEM_HUGE_ALWAYS:
-- 
2.39.0.314.g84b9a713c41-goog