Re: [REGRESSION] mremap() fails VMA expansion when VM_MIXEDMAP is set

[Vlastimil Babka (SUSE)]

On 10/23/26 04:31, Luiz Capitulino wrote:
> Growing a VM_MIXEDMAP VMA with mremap(), with or without MREMAP_MAYMOVE, fails
> with -ENOMEM when the address range after the VMA is free.
> 
> This regression was introduced in v6.1 by commit ca3d76b0aa80 ("mm: add merging
> after mremap resize"). This commit changed mremap() to use vma_merge() instead
> of vma_adjust() for the expansion case. However, vma_merge() returns NULL for
> a VMA that has any of the VM_SPECIAL flags set.

Hi,

6.1 was a rather long time ago. How was it discovered just now? I.e. what
use case broke by the change and prompted creating the reproducer?

Thanks,
Vlastimil

> The issue is also present in latest Linus tree, except that the code has been
> refactored. In the current tree, the path now is expand_vma_in_place() ->
> vma_merge_extend() -> vma_merge_new_range(), which performs the check:
> 
> 	/* Special VMAs are unmergeable, also if no prev/next. */
> 	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
> 	    (!prev && !next))
> 		return NULL;
> 
> This issue was originally reproduced with an external module that sets
> VM_MIXEDMAP on a VMA because it wants to insert pages with vm_insert_page().
> However, there's at least one in-tree reproducer: tcp_mmap().
> 
> Note that commit ca3d76b0aa80 introduced other VMA expansion regressions,
> but they were caught and fixed early:
> 
> - 6f12be792fde ("mm, mremap: fix mremap() expanding vma with addr inside vma")
> - d014cd7c1c35 ("mm, mremap: fix mremap() expanding for vma's with vm_ops->close()")
> 
> Below you'll find a reproducer and an attempted fix.
> 
> Reproducer
> ==========
> 
> This reproducer was written by an LLM and uses tcp_mmap(). It reproduces
> against v6.1 and latest Linus tree (2a08195e4db5). The issue is not
> reproduced on v6.0.
> 
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <unistd.h>
> #include <errno.h>
> #include <sys/mman.h>
> #include <sys/socket.h>
> 
> int main(void)
> {
>         int fd, pagesize;
>         void *p, *ret;
> 
>         fd = socket(AF_INET, SOCK_STREAM, 0);
>         if (fd < 0) {
>                 perror("socket");
>                 return 1;
>         }
> 
>         pagesize = getpagesize();
> 
>         /* tcp_mmap() sets VM_MIXEDMAP on this vma. */
>         p = mmap(NULL, 2 * pagesize, PROT_READ, MAP_SHARED, fd, 0);
>         if (p == MAP_FAILED) {
>                 perror("mmap");
>                 return 1;
>         }
> 
>         /* Drop the upper half, so growing back into it can happen in place. */
>         if (munmap((char *)p + pagesize, pagesize)) {
>                 perror("munmap");
>                 return 1;
>         }
> 
>         errno = 0;
>         ret = mremap(p, pagesize, 2 * pagesize, MREMAP_MAYMOVE);
>         if (ret == MAP_FAILED) {
>                 printf("BUG REPRODUCED errno=%d\n", errno);
>                 return 1;
>         }
> 
>         printf("NOT REPRODUCED\n");
> 
>         return 0;
> }
> 
> Sketch fix
> ==========
> 
> In the fix below, I started with the same idea as d014cd7c1c35: on v6.1 I
> made mremap() expand VM_MIXEDMAP VMAs the same way that commit did for 
> VMAs with vm_ops->close(). Then I adapted it to the current tree. It fixes
> the reproducer and I didn't get regressions in MM kselftests nor the
> tools/testing/vma test.
> 
> However, I'm not posting this as a fix because I'm unfamiliar with the VMA
> code so I'm not confident this is the right approach.
> 
> I really don't mind someone more familiar with the code taking this over.
> 
> diff --git a/mm/vma.c b/mm/vma.c
> index 6c35f0d775ab..ac9a0a8a4be9 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -851,9 +851,15 @@ static int commit_merge(struct vma_merge_struct *vmg)
>  	return 0;
>  }
>  
> -/* We can only remove VMAs when merging if they do not have a close hook. */
>  static bool can_merge_remove_vma(struct vm_area_struct *vma)
>  {
> +	if (vma_flags_test(&vma->flags, VMA_MIXEDMAP_BIT))
> +		return false;
> +
> +	/*
> +	 * We can only remove VMAs when merging if they do not have a
> +	 * close hook.
> +	 */
>  	return !vma->vm_ops || !vma->vm_ops->close;
>  }
>  
> @@ -1093,6 +1099,20 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(
>  	return NULL;
>  }
>  
> +static bool can_merge_vma(const struct vma_merge_struct *vmg)
> +{
> +	/* VMAs without prev/next are unmergeable */
> +	if (!vmg->prev && !vmg->next)
> +		return false;
> +
> +	/* We can't merge VMA_IO, VMA_DONTEXPAND or VMA_PFNMAP */
> +	if (vma_flags_test_any(&vmg->vma_flags, VMA_IO_BIT, VMA_DONTEXPAND_BIT,
> +			       VMA_PFNMAP_BIT))
> +		return false;
> +
> +	return true;
> +}
> +
>  /*
>   * vma_merge_new_range - Attempt to merge a new VMA into address space
>   *
> @@ -1152,9 +1172,7 @@ struct vm_area_struct *vma_merge_new_range(struct vma_merge_struct *vmg)
>  
>  	vmg->state = VMA_MERGE_NOMERGE;
>  
> -	/* Special VMAs are unmergeable, also if no prev/next. */
> -	if (vma_flags_test_any_mask(&vmg->vma_flags, VMA_SPECIAL_FLAGS) ||
> -	    (!prev && !next))
> +	if (!can_merge_vma(vmg))
>  		return NULL;
>  
>  	can_merge_left = can_vma_merge_left(vmg);
>