Re: [PATCH] mm/vma: update VMA iterator state on error in new VMA merge

From: Liam R. Howlett

Date: Thu Nov 12 2026 - 17:32:07 EST


On 26/10/27 11:25AM, Lorenzo Stoakes (ARM) wrote:

I'm a bit late on this one..

> Commit cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs") shared
> code between vma_merge_new_range() and vma_expand().
>
> However in doing so, it failed to correctly restore the VMA iterator when
> vma_expand() errors out.
>
> This is a problem when the immediately preceding VMA is mergeable and
> vmg->just_expand isn't specified, at which point the VMA iterator is moved
> to the previous VMA.
>
> (Note that vmg->just_expand is a special case for brk(), where the
> iterator will already be positioned at the VMA to be expanded.)
>
> Resolve things by restoring the VMA iterator on error. Use
> vma_iter_next_range() to do this, so it will correctly stop on the gap
> after the previous VMA, rather than skip to the next VMA after it (this is
> a new VMA merge, so is being applied at a gap).
>
> To isolate the error handling, return early when a merge isn't
> possible (what the vmg->target check was doing before), then handle the
> merge success/failure case explicitly.
>
> While here, update vma_expand() to jump to the nomem case when
> dup_anon_vma() fails, so vmg->state is correctly updated.
>
> This doesn't impact anything in practice, but is the correct course and
> since it's part of the error path that this patch addresses, it makes sense
> to update it also.
>
> Users are very unlikely to have observed the issue this patch addresses in
> practice, as it is near-impossible to fail an allocation for an anon_vma,
> but a process under memory pressure that receives a fatal signal could
> trigger it.
>
> This issue was originally reported by Li Jinyue for which I am grateful. He
> indicated this was uncovered by a private syzkaller instance.
>
> Fixes: cacded5e42b9 ("mm: avoid using vma_merge() for new VMAs")
> Cc: stable@xxxxxxxxxxxxxxx
> Reported-by: Li Jinyue <jinyue.li@xxxxxxxxxxxxxxxxx>
> Closes: https://lore.kernel.org/all/20261025093921.1011927-1-jinyue.li@xxxxxxxxxxxxxxxxx/
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
> ---
> mm/vma.c | 28 ++++++++++++++++++----------
> 1 file changed, 18 insertions(+), 10 deletions(-)
>
> diff --git a/mm/vma.c b/mm/vma.c
> index af8e65ada23b..d416ba58eb4c 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -1141,6 +1141,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> struct vm_area_struct *next = vmg->next;
> unsigned long end = vmg->end;
> bool can_merge_left, can_merge_right;
> + bool move_iter_back = false;

The name move_iter_back is a bit confusing because you use it to move it
to the next entry. I get what you mean, move it back to where it was, not
go to prev, but the name is a bit off when used to go to the next entry.

reposition_iter, maybe?

>
> mmap_assert_write_locked(vmg->mm);
> VM_WARN_ON_VMG(vmg->middle, vmg);
> @@ -1182,23 +1183,30 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> vmg->end = end;
>
> /* In expand-only case we are already positioned at prev. */
> - if (!vmg->just_expand) {
> - /* Equivalent to going to the previous range. */
> - vma_prev(vmg->vmi);
> - }
> + if (!vmg->just_expand)
> + move_iter_back = true;
> }
>
> + if (!can_merge_left && !can_merge_right)
> + return NULL;
> +
> + if (move_iter_back)
> + vma_prev(vmg->vmi);
> +
> /*
> * Now try to expand adjacent VMA(s). This takes care of removing the
> * following VMA if we have VMAs on both sides.
> */
> - if (vmg->target && !vma_expand(vmg)) {
> - khugepaged_enter_vma(vmg->target, vmg->vm_flags);
> - vmg->state = VMA_MERGE_SUCCESS;
> - return vmg->target;
> + if (vma_expand(vmg)) {
> + /* Error occurred, restore iterator. */
> + if (move_iter_back)
> + vma_iter_next_range(vmg->vmi);
> + return NULL;
> }
>
> - return NULL;
> + khugepaged_enter_vma(vmg->target, vmg->vm_flags);
> + vmg->state = VMA_MERGE_SUCCESS;
> + return vmg->target;
> }
>
> /*
> @@ -1288,7 +1296,7 @@ int vma_expand(struct vma_merge_struct *vmg)
> if (!ret && vmg->copied_from)
> ret = dup_anon_vma(target, vmg->copied_from, &anon_dup);
> if (ret)
> - return ret;
> + goto nomem;
>
> if (remove_next) {
> vma_start_write(next);
>
> ---
> base-commit: 1eb36316305767b54fac2cd40b88b3499791bdf2
> change-id: 20261027-fix-new-merge-bug-71c2200d40fb
>
> Cheers,
> --
> Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
>