[v4 PATCH 0/2] Fix false negative of shmem vma's THP eligibility

From: Yang Shi
Date: Wed Jul 17 2019 - 17:59:44 EST

The commit 7635d9cbe832 ("mm, thp, proc: report THP eligibility for each
vma") introduced THPeligible bit for processes' smaps. But, when checking
the eligibility for shmem vma, __transparent_hugepage_enabled() is
called to override the result from shmem_huge_enabled(). It may result
in the anonymous vma's THP flag override shmem's. For example, running a
simple test which create THP for shmem, but with anonymous THP disabled,
when reading the process's smaps, it may show:

7fc92ec00000-7fc92f000000 rw-s 00000000 00:14 27764 /dev/shm/test
Size: 4096 kB
ShmemPmdMapped: 4096 kB
THPeligible: 0

And, /proc/meminfo does show THP allocated and PMD mapped too:

ShmemHugePages: 4096 kB
ShmemPmdMapped: 4096 kB

This doesn't make too much sense. The shmem objects should be treated
separately from anonymous THP. Calling shmem_huge_enabled() with checking
MMF_DISABLE_THP sounds good enough. And, we could skip stack and
dax vma check since we already checked if the vma is shmem already.

The transhuge_vma_suitable() is needed to check vma, but it was only
available for shmem THP. The patch 1/2 makes it available for all kind of
THPs and does some code duplication cleanup, so it is made a separate patch.

v4: * Moved transhuge_vma_suitable() to include/linux/huge_mm.h and
regroup some functions in linux/include/mm.h. Per Hugh Dickins.
* Added Hughâs Acked-by to patch 2/2.
v3: * Check if vma is suitable for allocating THP per Hugh Dickins.
* Fixed smaps output alignment and documentation per Hugh Dickins.
v2: * Check VM_NOHUGEPAGE per Michal Hocko.

Yang Shi (2):
mm: thp: make transhuge_vma_suitable available for anonymous THP
mm: thp: fix false negative of shmem vma's THP eligibility

Documentation/filesystems/proc.txt | 4 ++--
fs/proc/task_mmu.c | 3 ++-
include/linux/huge_mm.h | 23 +++++++++++++++++++++++
include/linux/mm.h | 34 +++++++++++++++++-----------------
mm/huge_memory.c | 11 ++++++++---
mm/memory.c | 13 -------------
mm/shmem.c | 3 +++
7 files changed, 55 insertions(+), 36 deletions(-)