[PATCH for 7.4-rc2+ 5/7] sparc64: mm: fix thp_pte_count accounting around pmdp_invalidate()

[Zi Yan]

thp_pte_count is maintained in __set_pmd_acct() whenever _PAGE_PMD_HUGE
flips. pmdp_invalidate() only clears _PAGE_VALID, so it does not flip
that bit, but it also decrements thp_pte_count on its own, because THP
split replaces the invalidated PMD with a page table via pmd_populate(),
which does no accounting.

That is only right if the invalidated PMD is never set back. Several
callers invalidate a PMD and then install a valid huge PMD again with
set_pmd_at(): MADV_FREE, MADV_COLD clearing the young bit,
page_mkclean_one() and change_huge_pmd(). The valid PMD is not counted
again, as _PAGE_PMD_HUGE did not flip, so each such call loses one count.

Count only valid huge PMDs instead: in __set_pmd_acct(), adjust the
count when an entry starts or stops being both _PAGE_PMD_HUGE and
_PAGE_VALID, and drop the separate decrement from pmdp_invalidate(),
whose pmdp_establish() now accounts for the invalidation itself. An
invalidated PMD that is set again, cleared or replaced by a page table
then stays balanced. A PROT_NONE THP is no longer counted either, which
suits the TSB sizing the count is used for, as only valid entries are
inserted into the TSB.

Fixes: 1e953d846ac0 ("sparc64 mm: Fix more TSB sizing issues")
Assisted-by: LLM
Signed-off-by: Zi Yan <ziy@xxxxxxxxxx>
To: "David S. Miller" <davem@xxxxxxxxxxxxx>
To: Andreas Larsson <andreas@xxxxxxxxxxx>
Cc: sparclinux@xxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx
---
 arch/sparc/mm/tlb.c | 26 ++++++++++++--------------
 1 file changed, 12 insertions(+), 14 deletions(-)

diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index 6d9dd5eb13287..0a941d497525f 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -176,13 +176,23 @@ static void tlb_batch_pmd_scan(struct mm_struct *mm, unsigned long vaddr,
 }
 
 
+/*
+ * Only valid huge PMDs are counted: an invalidated or PROT_NONE PMD gives no
+ * access and never goes into the TSB.
+ */
+static bool pmd_huge_valid(pmd_t pmd)
+{
+	return (pmd_val(pmd) & (_PAGE_PMD_HUGE | _PAGE_VALID)) ==
+	       (_PAGE_PMD_HUGE | _PAGE_VALID);
+}
+
 static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr,
 			   pmd_t orig, pmd_t pmd)
 {
 	if (mm == &init_mm)
 		return;
 
-	if ((pmd_val(pmd) ^ pmd_val(orig)) & _PAGE_PMD_HUGE) {
+	if (pmd_huge_valid(pmd) != pmd_huge_valid(orig)) {
 		/*
 		 * Note that this routine only sets pmds for THP pages.
 		 * Hugetlb pages are handled elsewhere.  We need to check
@@ -191,7 +201,7 @@ static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr,
 		 * for TSB entries.  So, huge zero page counts go into
 		 * hugetlb_pte_count.
 		 */
-		if (pmd_val(pmd) & _PAGE_PMD_HUGE) {
+		if (pmd_huge_valid(pmd)) {
 			if (is_huge_zero_pmd(pmd))
 				mm->context.hugetlb_pte_count++;
 			else
@@ -250,9 +260,6 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
 	return old;
 }
 
-/*
- * This routine is only called when splitting a THP
- */
 pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 		     pmd_t *pmdp)
 {
@@ -263,15 +270,6 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
 	old = pmdp_establish(vma, address, pmdp, entry);
 	flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
 
-	/*
-	 * set_pmd_at() will not be called in a way to decrement
-	 * thp_pte_count when splitting a THP, so do it now.
-	 * Sanity check pmd before doing the actual decrement.
-	 */
-	if ((pmd_val(entry) & _PAGE_PMD_HUGE) &&
-	    !is_huge_zero_pmd(entry))
-		(vma->vm_mm)->context.thp_pte_count--;
-
 	return old;
 }
 

-- 
2.53.0