Re: [PATCH v17 11/23] mm: pagewalk: Add p4d_entry() and pgd_entry()

From: Steven Price
Date: Fri Dec 20 2019 - 10:35:51 EST

Next message: Christopher Lameter: "Re: Percpu variables, benchmarking, and performance weirdness"
Previous message: Joakim Zhang: "RE: [PATCH V3 2/2] drivers/irqchip: add NXP INTMUX interrupt multiplexer support"
In reply to: Thomas HellstrÃm (VMware): "Re: [PATCH v17 11/23] mm: pagewalk: Add p4d_entry() and pgd_entry()"
Next in thread: Steven Price: "[PATCH v17 12/23] mm: pagewalk: Allow walking without vma"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On 19/12/2019 14:25, Thomas HellstrÃm (VMware) wrote:
> Hi, Steven,
>
>
> On 12/18/19 5:23 PM, Steven Price wrote:
>> pgd_entry() and pud_entry() were removed by commit 0b1fbfe50006c410
>> ("mm/pagewalk: remove pgd_entry() and pud_entry()") because there were
>> no users. We're about to add users so reintroduce them, along with
>> p4d_entry() as we now have 5 levels of tables.
>>
>> Note that commit a00cc7d9dd93d66a ("mm, x86: add support for
>> PUD-sized transparent hugepages") already re-added pud_entry() but with
>> different semantics to the other callbacks. This commit reverts the
>> semantics back to match the other callbacks.
>>
>> To support hmm.c which now uses the new semantics of pud_entry() a new
>> member ('action') of struct mm_walk is added which allows the callbacks
>> to either descend (ACTION_SUBTREE, the default), skip (ACTION_CONTINUE)
>> or repeat the callback (ACTION_AGAIN). hmm.c is then updated to call
>> pud_trans_huge_lock() itself and make use of the splitting/retry logic
>> of the core code.
>>
>> After this change pud_entry() is called for all entries, not just
>> transparent huge pages.
>>
>> Signed-off-by: Steven Price <steven.price@xxxxxxx>
>
> I have a couple of comments that are actually mostly related to
> pre-existing bugs, but that affect code that are touched by this patch.
>
> Perhaps this is most cleanly addressed by a follow-up patch. Up to you
> to decide.
>
> Since these problems are pre-existing,
>
> Reviewed-by: Thomas Hellstrom <thellstrom@xxxxxxxxxx>

Thanks, I'll send out an extra patch with these changes.

Steve

> Thanks,
>
> Thomas
>
>
>> ---
>> Â include/linux/pagewalk.h | 34 ++++++++++++++++++++-----
>> Â mm/hmm.cÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | 55 ++++++++++++++++++++++------------------
>> Â mm/pagewalk.cÂÂÂÂÂÂÂÂÂÂÂ | 50 +++++++++++++++++++++++++-----------
>> Â 3 files changed, 94 insertions(+), 45 deletions(-)
>>
>> diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h
>> index 6ec82e92c87f..aa6a0b63964e 100644
>> --- a/include/linux/pagewalk.h
>> +++ b/include/linux/pagewalk.h
>> @@ -8,15 +8,15 @@ struct mm_walk;
>> Â Â /**
>> ÂÂ * mm_walk_ops - callbacks for walk_page_range
>> - * @pud_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PUD
>> (2nd-level) entry
>> - *ÂÂÂÂÂÂÂÂÂÂÂ this handler should only handle pud_trans_huge() puds.
>> - *ÂÂÂÂÂÂÂÂÂÂÂ the pmd_entry or pte_entry callbacks will be used for
>> - *ÂÂÂÂÂÂÂÂÂÂÂ regular PUDs.
>> - * @pmd_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PMD
>> (3rd-level) entry
>> + * @pgd_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PGD
>> (top-level) entry
>> + * @p4d_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty P4D entry
>> + * @pud_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PUD entry
>> + * @pmd_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PMD entry
>> ÂÂ *ÂÂÂÂÂÂÂÂÂÂÂ this handler is required to be able to handle
>> ÂÂ *ÂÂÂÂÂÂÂÂÂÂÂ pmd_trans_huge() pmds.Â They may simply choose to
>> ÂÂ *ÂÂÂÂÂÂÂÂÂÂÂ split_huge_page() instead of handling it explicitly.
>> - * @pte_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PTE
>> (4th-level) entry
>> + * @pte_entry:ÂÂÂÂÂÂÂ if set, called for each non-empty PTE
>> (lowest-level)
>> + *ÂÂÂÂÂÂÂÂÂÂÂ entry
>> ÂÂ * @pte_hole:ÂÂÂÂÂÂÂ if set, called for each hole at all levels
>> ÂÂ * @hugetlb_entry:ÂÂÂ if set, called for each hugetlb entry
>> ÂÂ * @test_walk:ÂÂÂÂÂÂÂ caller specific callback function to determine
>> whether
>> @@ -27,8 +27,15 @@ struct mm_walk;
>> ÂÂ * @pre_vma:ÂÂÂÂÂÂÂÂÂÂÂ if set, called before starting walk on a
>> non-null vma.
>> ÂÂ * @post_vma:ÂÂÂÂÂÂÂÂÂÂ if set, called after a walk on a non-null
>> vma, provided
>> ÂÂ *ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ that @pre_vma and the vma walk succeeded.
>> + *
>> + * p?d_entry callbacks are called even if those levels are folded on a
>> + * particular architecture/configuration.
>> ÂÂ */
>> Â struct mm_walk_ops {
>> +ÂÂÂ int (*pgd_entry)(pgd_t *pgd, unsigned long addr,
>> +ÂÂÂÂÂÂÂÂÂÂÂÂ unsigned long next, struct mm_walk *walk);
>> +ÂÂÂ int (*p4d_entry)(p4d_t *p4d, unsigned long addr,
>> +ÂÂÂÂÂÂÂÂÂÂÂÂ unsigned long next, struct mm_walk *walk);
>> ÂÂÂÂÂ int (*pud_entry)(pud_t *pud, unsigned long addr,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ unsigned long next, struct mm_walk *walk);
>> ÂÂÂÂÂ int (*pmd_entry)(pmd_t *pmd, unsigned long addr,
>> @@ -47,11 +54,25 @@ struct mm_walk_ops {
>> ÂÂÂÂÂ void (*post_vma)(struct mm_walk *walk);
>> Â };
>> Â +/*
>> + * Action for pud_entry / pmd_entry callbacks.
>> + * ACTION_SUBTREE is the default
>> + */
>> +enum page_walk_action {
>> +ÂÂÂ /* Descend to next level, splitting huge pages if needed and
>> possible */
>> +ÂÂÂ ACTION_SUBTREE = 0,
>> +ÂÂÂ /* Continue to next entry at this level (ignoring any subtree) */
>> +ÂÂÂ ACTION_CONTINUE = 1,
>> +ÂÂÂ /* Call again for this entry */
>> +ÂÂÂ ACTION_AGAIN = 2
>> +};
>> +
>> Â /**
>> ÂÂ * mm_walk - walk_page_range data
>> ÂÂ * @ops:ÂÂÂ operation to call during the walk
>> ÂÂ * @mm:ÂÂÂÂÂÂÂ mm_struct representing the target process of page
>> table walk
>> ÂÂ * @vma:ÂÂÂ vma currently walked (NULL if walking outside vmas)
>> + * @action:ÂÂÂ next action to perform (see enum page_walk_action)
>> ÂÂ * @private:ÂÂÂ private data for callbacks' usage
>> ÂÂ *
>> ÂÂ * (see the comment on walk_page_range() for more details)
>> @@ -60,6 +81,7 @@ struct mm_walk {
>> ÂÂÂÂÂ const struct mm_walk_ops *ops;
>> ÂÂÂÂÂ struct mm_struct *mm;
>> ÂÂÂÂÂ struct vm_area_struct *vma;
>> +ÂÂÂ enum page_walk_action action;
>> ÂÂÂÂÂ void *private;
>> Â };
>> Â diff --git a/mm/hmm.c b/mm/hmm.c
>> index d379cb6496ae..05241c82e05c 100644
>> --- a/mm/hmm.c
>> +++ b/mm/hmm.c
>> @@ -477,20 +477,30 @@ static int hmm_vma_walk_pud(pud_t *pudp,
>> unsigned long start, unsigned long end,
>> ÂÂÂÂÂ unsigned long addr = start, next;
>> ÂÂÂÂÂ pmd_t *pmdp;
>> ÂÂÂÂÂ pud_t pud;
>> -ÂÂÂ int ret;
>> +ÂÂÂ int ret = 0;
>> +ÂÂÂ spinlock_t *ptl = pud_trans_huge_lock(pudp, walk->vma);
>> +
>> +ÂÂÂ if (!ptl)
>> +ÂÂÂÂÂÂÂ return 0;
>
> Since if we didn't get the PTL here, the pud might be unstable in which
> case we want to retry:
>
> if (!ptl) {
> ÂÂÂÂif (pud_trans_unstable(pudp))
> ÂÂÂÂÂÂÂ walk->action = ACTION_AGAIN;
> ÂÂÂÂreturn 0;
> }
>
>
>> +
>> +ÂÂÂ /* Normally we don't want to split the huge page */
>> +ÂÂÂ walk->action = ACTION_CONTINUE;
>> Â -again:
>> ÂÂÂÂÂ pud = READ_ONCE(*pudp);
>
> We have the lock now, so READ_ONCE could probably be a simple dereference.
>
>> -ÂÂÂ if (pud_none(pud))
>> -ÂÂÂÂÂÂÂ return hmm_vma_walk_hole(start, end, walk);
>> +ÂÂÂ if (pud_none(pud)) {
>> +ÂÂÂÂÂÂÂ ret = hmm_vma_walk_hole(start, end, walk);
>> +ÂÂÂÂÂÂÂ goto out_unlock;
>> +ÂÂÂ }
>
> pud_trans_huge_lock() successful return means pud_none() is always false.
>
>
>> Â ÂÂÂÂÂ if (pud_huge(pud) && pud_devmap(pud)) {
>> ÂÂÂÂÂÂÂÂÂ unsigned long i, npages, pfn;
>> ÂÂÂÂÂÂÂÂÂ uint64_t *pfns, cpu_flags;
>> ÂÂÂÂÂÂÂÂÂ bool fault, write_fault;
>> Â -ÂÂÂÂÂÂÂ if (!pud_present(pud))
>> -ÂÂÂÂÂÂÂÂÂÂÂ return hmm_vma_walk_hole(start, end, walk);
>> +ÂÂÂÂÂÂÂ if (!pud_present(pud)) {
>> +ÂÂÂÂÂÂÂÂÂÂÂ ret = hmm_vma_walk_hole(start, end, walk);
>> +ÂÂÂÂÂÂÂÂÂÂÂ goto out_unlock;
>> +ÂÂÂÂÂÂÂ }
>
> Can't see !pud_present happening either after a succesful
> pud_trans_huge_lock().
>
>> Â ÂÂÂÂÂÂÂÂÂ i = (addr - range->start) >> PAGE_SHIFT;
>> ÂÂÂÂÂÂÂÂÂ npages = (end - addr) >> PAGE_SHIFT;
>> @@ -499,16 +509,20 @@ static int hmm_vma_walk_pud(pud_t *pudp,
>> unsigned long start, unsigned long end,
>> ÂÂÂÂÂÂÂÂÂ cpu_flags = pud_to_hmm_pfn_flags(range, pud);
>> ÂÂÂÂÂÂÂÂÂ hmm_range_need_fault(hmm_vma_walk, pfns, npages,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ cpu_flags, &fault, &write_fault);
>> -ÂÂÂÂÂÂÂ if (fault || write_fault)
>> -ÂÂÂÂÂÂÂÂÂÂÂ return hmm_vma_walk_hole_(addr, end, fault,
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ write_fault, walk);
>> +ÂÂÂÂÂÂÂ if (fault || write_fault) {
>> +ÂÂÂÂÂÂÂÂÂÂÂ ret = hmm_vma_walk_hole_(addr, end, fault,
>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ write_fault, walk);
>> +ÂÂÂÂÂÂÂÂÂÂÂ goto out_unlock;
>> +ÂÂÂÂÂÂÂ }
>> Â ÂÂÂÂÂÂÂÂÂ pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
>> ÂÂÂÂÂÂÂÂÂ for (i = 0; i < npages; ++i, ++pfn) {
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ hmm_vma_walk->pgmap = get_dev_pagemap(pfn,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ hmm_vma_walk->pgmap);
>> -ÂÂÂÂÂÂÂÂÂÂÂ if (unlikely(!hmm_vma_walk->pgmap))
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ return -EBUSY;
>> +ÂÂÂÂÂÂÂÂÂÂÂ if (unlikely(!hmm_vma_walk->pgmap)) {
>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ret = -EBUSY;
>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ goto out_unlock;
>> +ÂÂÂÂÂÂÂÂÂÂÂ }
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ pfns[i] = hmm_device_entry_from_pfn(range, pfn) |
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ cpu_flags;
>> ÂÂÂÂÂÂÂÂÂ }
>> @@ -517,22 +531,15 @@ static int hmm_vma_walk_pud(pud_t *pudp,
>> unsigned long start, unsigned long end,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ hmm_vma_walk->pgmap = NULL;
>> ÂÂÂÂÂÂÂÂÂ }
>> ÂÂÂÂÂÂÂÂÂ hmm_vma_walk->last = end;
>> -ÂÂÂÂÂÂÂ return 0;
>> +ÂÂÂÂÂÂÂ goto out_unlock;
>> ÂÂÂÂÂ }
>> Â -ÂÂÂ split_huge_pud(walk->vma, pudp, addr);
>> -ÂÂÂ if (pud_none(*pudp))
>> -ÂÂÂÂÂÂÂ goto again;
>> +ÂÂÂ /* Ask for the PUD to be split */
>> +ÂÂÂ walk->action = ACTION_SUBTREE;
>> Â -ÂÂÂ pmdp = pmd_offset(pudp, addr);
>> -ÂÂÂ do {
>> -ÂÂÂÂÂÂÂ next = pmd_addr_end(addr, end);
>> -ÂÂÂÂÂÂÂ ret = hmm_vma_walk_pmd(pmdp, addr, next, walk);
>> -ÂÂÂÂÂÂÂ if (ret)
>> -ÂÂÂÂÂÂÂÂÂÂÂ return ret;
>> -ÂÂÂ } while (pmdp++, addr = next, addr != end);
>> -
>> -ÂÂÂ return 0;
>> +out_unlock:
>> +ÂÂÂ spin_unlock(ptl);
>> +ÂÂÂ return ret;
>> Â }
>> Â #else
>> Â #define hmm_vma_walk_pudÂÂÂ NULL
>> diff --git a/mm/pagewalk.c b/mm/pagewalk.c
>> index ea0b9e606ad1..690af44609e2 100644
>> --- a/mm/pagewalk.c
>> +++ b/mm/pagewalk.c
>> @@ -46,6 +46,9 @@ static int walk_pmd_range(pud_t *pud, unsigned long
>> addr, unsigned long end,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ continue;
>> ÂÂÂÂÂÂÂÂÂ }
>> +
>> +ÂÂÂÂÂÂÂ walk->action = ACTION_SUBTREE;
>> +
>> ÂÂÂÂÂÂÂÂÂ /*
>> ÂÂÂÂÂÂÂÂÂÂ * This implies that each ->pmd_entry() handler
>> ÂÂÂÂÂÂÂÂÂÂ * needs to know about pmd_trans_huge() pmds
>> @@ -55,16 +58,21 @@ static int walk_pmd_range(pud_t *pud, unsigned
>> long addr, unsigned long end,
>> ÂÂÂÂÂÂÂÂÂ if (err)
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
>> Â +ÂÂÂÂÂÂÂ if (walk->action == ACTION_AGAIN)
>> +ÂÂÂÂÂÂÂÂÂÂÂ goto again;
>> +
>> ÂÂÂÂÂÂÂÂÂ /*
>> ÂÂÂÂÂÂÂÂÂÂ * Check this here so we only break down trans_huge
>> ÂÂÂÂÂÂÂÂÂÂ * pages when we _need_ to
>> ÂÂÂÂÂÂÂÂÂÂ */
>> -ÂÂÂÂÂÂÂ if (!ops->pte_entry)
>> +ÂÂÂÂÂÂÂ if (walk->action == ACTION_CONTINUE ||
>> +ÂÂÂÂÂÂÂÂÂÂÂ !(ops->pte_entry))
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ continue;
>> Â ÂÂÂÂÂÂÂÂÂ split_huge_pmd(walk->vma, pmd, addr);
>> ÂÂÂÂÂÂÂÂÂ if (pmd_trans_unstable(pmd))
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ goto again;
>> +
>> ÂÂÂÂÂÂÂÂÂ err = walk_pte_range(pmd, addr, next, walk);
>> ÂÂÂÂÂÂÂÂÂ if (err)
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
>> @@ -93,24 +101,25 @@ static int walk_pud_range(p4d_t *p4d, unsigned
>> long addr, unsigned long end,
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ continue;
>> ÂÂÂÂÂÂÂÂÂ }
>> Â -ÂÂÂÂÂÂÂ if (ops->pud_entry) {
>> -ÂÂÂÂÂÂÂÂÂÂÂ spinlock_t *ptl = pud_trans_huge_lock(pud, walk->vma);
>> +ÂÂÂÂÂÂÂ walk->action = ACTION_SUBTREE;
>> Â -ÂÂÂÂÂÂÂÂÂÂÂ if (ptl) {
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ err = ops->pud_entry(pud, addr, next, walk);
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ spin_unlock(ptl);
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if (err)
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
>> -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ continue;
>> -ÂÂÂÂÂÂÂÂÂÂÂ }
>> -ÂÂÂÂÂÂÂ }
>> +ÂÂÂÂÂÂÂ if (ops->pud_entry)
>> +ÂÂÂÂÂÂÂÂÂÂÂ err = ops->pud_entry(pud, addr, next, walk);
>> +ÂÂÂÂÂÂÂ if (err)
>> +ÂÂÂÂÂÂÂÂÂÂÂ break;
>> +
>> +ÂÂÂÂÂÂÂ if (walk->action == ACTION_AGAIN)
>> +ÂÂÂÂÂÂÂÂÂÂÂ goto again;
>> +
>> +ÂÂÂÂÂÂÂ if (walk->action == ACTION_CONTINUE ||
>> +ÂÂÂÂÂÂÂÂÂÂÂ !(ops->pmd_entry || ops->pte_entry))
>> +ÂÂÂÂÂÂÂÂÂÂÂ continue;
>> Â ÂÂÂÂÂÂÂÂÂ split_huge_pud(walk->vma, pud, addr);
>> ÂÂÂÂÂÂÂÂÂ if (pud_none(*pud))
>
> Here we should really have
>
> if (pud_trans_unstable(pud))
>
> Thanks,
>
> Thomas
>
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Next message: Christopher Lameter: "Re: Percpu variables, benchmarking, and performance weirdness"
Previous message: Joakim Zhang: "RE: [PATCH V3 2/2] drivers/irqchip: add NXP INTMUX interrupt multiplexer support"
In reply to: Thomas HellstrÃm (VMware): "Re: [PATCH v17 11/23] mm: pagewalk: Add p4d_entry() and pgd_entry()"
Next in thread: Steven Price: "[PATCH v17 12/23] mm: pagewalk: Allow walking without vma"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]