Re: [REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Luiz Capitulino]

On 10/23/26 3:16 AM, Vlastimil Babka (SUSE) wrote:
> On 10/23/26 04:31, Luiz Capitulino wrote:
> > Growing a VM_MIXEDMAP VMA with mremap(), with or without MREMAP_MAYMOVE, fails
> > with -ENOMEM when the address range after the VMA is free.
> >
> > This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
> > after mremap resize"). This commit changed mremap() to use vma_merge() instead
> > of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
> > a VMA that has any of the VM_SPECIAL flags set.
>
> Hi,
>
> 6.1 was a rather long time ago. How was it discovered just now? I.e. what
> use case broke by the change and prompted creating the reproducer?

The use-case that broke was an application running on an older kernel
and accessing a GPFS volume/FS. I was told that GPFS sets VM_MIXEDMAP on
a VMA because it inserts pages in the process address space with
vm_insert_page(). I then asked an LLM to check if there was anything in
tree that would reproduce that, it found the tcp_mmap() case and created
the attached reproducer. Btw, the LLM also told me there are more cases
in-tree that can reproduce this (but they require special hardware or
setup so I didn't investigate).

> Thanks,
> Vlastimil
>
> > The issue is also present in latest Linus tree, except that the code has been
> > refactored. In the current tree, the path now is expand_vma_in_place() ->
> > vma_merge_extend() -> vma_merge_new_range(), which performs the check:
> >
> > 	/* Special VMAs are unmergeable, also if no prev/next. */
> > 	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
> > 	    (!prev && !next))
> > 		return NULL;
> >
> > This issue was originally reproduced with an external module that sets
> > VM_MIXEDMAP on a VMA because it wants to insert pages with vm_insert_page().
> > However, there's at least one in-tree reproducer: tcp_mmap().
> >
> > Note that commit ca3d76b0aa80 introduced other VMA expansion regressions,
> > but they were caught and fixed early:
> >
> > - 6f12be792fde ("mm, mremap: fix mremap() expanding vma with addr inside vma")
> > - d014cd7c1c35 ("mm, mremap: fix mremap() expanding for vma's with vm_ops->close()")
> >
> > Below you'll find a reproducer and an attempted fix.
> >
> > Reproducer
> > ==========
> >
> > This reproducer was written by an LLM and uses tcp_mmap(). It reproduces
> > against v6.1 and latest Linus tree (2a08195e4db5). The issue is not
> > reproduced on v6.0.
> >
> > #define _GNU_SOURCE
> > #include <stdio.h>
> > #include <unistd.h>
> > #include <errno.h>
> > #include <sys/mman.h>
> > #include <sys/socket.h>
> >
> > int main(void)
> > {
> >          int fd, pagesize;
> >          void *p, *ret;
> >
> >          fd = socket(AF_INET, SOCK_STREAM, 0);
> >          if (fd < 0) {
> >                  perror("socket");
> >                  return 1;
> >          }
> >
> >          pagesize = getpagesize();
> >
> >          /* tcp_mmap() sets VM_MIXEDMAP on this vma. */
> >          p = mmap(NULL, 2 * pagesize, PROT_READ, MAP_SHARED, fd, 0);
> >          if (p == MAP_FAILED) {
> >                  perror("mmap");
> >                  return 1;
> >          }
> >
> >          /* Drop the upper half, so growing back into it can happen in place. */
> >          if (munmap((char *)p + pagesize, pagesize)) {
> >                  perror("munmap");
> >                  return 1;
> >          }
> >
> >          errno = 0;
> >          ret = mremap(p, pagesize, 2 * pagesize, MREMAP_MAYMOVE);
> >          if (ret == MAP_FAILED) {
> >                  printf("BUG REPRODUCED errno=%d\n", errno);
> >                  return 1;
> >          }
> >
> >          printf("NOT REPRODUCED\n");
> >
> >          return 0;
> > }
> >
> > Sketch fix
> > ==========
> >
> > In the fix below, I started with the same idea as d014cd7c1c35: on v6.1 I
> > made mremap() expand VM_MIXEDMAP VMAs the same way that commit did for
> > VMAs with vm_ops->close(). Then I adapted it to the current tree. It fixes
> > the reproducer and I didn't get regressions in MM kselftests nor the
> > tools/testing/vma test.
> >
> > However, I'm not posting this as a fix because I'm unfamiliar with the VMA
> > code so I'm not confident this is the right approach.
> >
> > I really don't mind someone more familiar with the code taking this over.
> >
> > diff --git a/mm/vma.c b/mm/vma.c
> > index 6c35f0d775ab..ac9a0a8a4be9 100644
> > --- a/mm/vma.c
> > +++ b/mm/vma.c
> > @@ -851,9 +851,15 @@ static int commit_merge(struct vma_merge_struct *vmg)
> >   	return 0;
> >   }
> >   
> > -/* We can only remove VMAs when merging if they do not have a close hook. */
> >   static bool can_merge_remove_vma(struct vm_area_struct *vma)
> >   {
> > +	if (vma_flags_test(&vma->flags, VMA_MIXEDMAP_BIT))
> > +		return false;
> > +
> > +	/*
> > +	 * We can only remove VMAs when merging if they do not have a
> > +	 * close hook.
> > +	 */
> >   	return !vma->vm_ops || !vma->vm_ops->close;
> >   }
> >   
> > @@ -1093,6 +1099,20 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(
> >   	return NULL;
> >   }
> >   
> > +static bool can_merge_vma(const struct vma_merge_struct *vmg)
> > +{
> > +	/* VMAs without prev/next are unmergeable */
> > +	if (!vmg->prev && !vmg->next)
> > +		return false;
> > +
> > +	/* We can't merge VMA_IO, VMA_DONTEXPAND or VMA_PFNMAP */
> > +	if (vma_flags_test_any(&vmg->vma_flags, VMA_IO_BIT, VMA_DONTEXPAND_BIT,
> > +			       VMA_PFNMAP_BIT))
> > +		return false;
> > +
> > +	return true;
> > +}
> > +
> >   /*
> >    * vma_merge_new_range - Attempt to merge a new VMA into address space
> >    *
> > @@ -1152,9 +1172,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
> >   
> >   	vmg->state = VMA_MERGE_NOMERGE;
> >   
> > -	/* Special VMAs are unmergeable, also if no prev/next. */
> > -	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
> > -	    (!prev && !next))
> > +	if (!can_merge_vma(vmg))
> >   		return NULL;
> >   
> >   	can_merge_left = can_vma_merge_left(vmg);