Re: [PATCH RFC] mm/madvise: introduce MADV_POPULATE to prefault/prealloc memory

[David Hildenbrand]

> Am 19.02.2021 um 20:23 schrieb Peter Xu <peterx@xxxxxxxxxx>:
> 
> On Fri, Feb 19, 2021 at 06:13:47PM +0100, David Hildenbrand wrote:
>>> On 19.02.21 17:31, Peter Xu wrote:
>>> On Fri, Feb 19, 2021 at 09:20:16AM +0100, David Hildenbrand wrote:
>>>> On 18.02.21 23:59, Peter Xu wrote:
>>>>> Hi, David,
>>>>> 
>>>>> On Wed, Feb 17, 2021 at 04:48:44PM +0100, David Hildenbrand wrote:
>>>>>> When we manage sparse memory mappings dynamically in user space - also
>>>>>> sometimes involving MADV_NORESERVE - we want to dynamically populate/
>>>>>> discard memory inside such a sparse memory region. Example users are
>>>>>> hypervisors (especially implementing memory ballooning or similar
>>>>>> technologies like virtio-mem) and memory allocators. In addition, we want
>>>>>> to fail in a nice way if populating does not succeed because we are out of
>>>>>> backend memory (which can happen easily with file-based mappings,
>>>>>> especially tmpfs and hugetlbfs).
> 
> [1]
> 
>>> E.g., can we simply ask the kernel "how much memory this process can still
>>> allocate", then get a number out of it?  I'm not sure whether it can be done
>> 
>> Anything like that is completely racy and unreliable.
> 
> The failure path won't be racy imho - If we can detect current process doesn't
> have enough memory budget, it'll be more efficient to fail even before trying
> to populate any memory and then drop part of them again.
> 
> But I see your point - indeed it's good to guarantee the guest won't crash at
> any point of further guest side memory access.
> 
> Another question: can the user actually specify arbitrary max-length for the
> virtio-mem device (which decides the maximum memory this device could possibly
> consume)?  I thought we should check that first before realizing the device and
> we really shouldn't fail any guest memory access if that check passed. Feel
> free to correct me.

Max-length is currently limited by the mmap() we‘re allowed to create. With MAP_NORESERVE this can be big (not merged yet).

Checking max-lenght at initialization time does not make too much sense. Just imagine shrinking/relocating other VMs so you can grow this VM further. Or migrating the VM to another machine where you might grow it further.

The ultimate goal is to adjust the mapping size dynamically on demand, but that‘s stuff for the future as it turns out complicated. For example, hugetlbfs VMAs cannot be merged yet (although I think it shouldn‘t be too hard to implement).

The short term approach is only exposing a small window of the bigger mmap to the guest.

>> 
>> That would be kind of weird. I'd assume the reservation gets properly done
>> during fork() - just like for VM_ACCOUNT.
> 
> AFAIK VM_ACCOUNT is never applied for hugetlbfs.  Neither do I know any
> accounting done for hugetlbfs during fork(), if not taking the pinned pages
> into account - that is definitely a special case.
> 

Yes, it isn‘t - I meant „like“ as in „similar to swap reservation“.

>> 
>>> However that's definitely not the case for QEMU since QEMU won't work at all as
>>> late as that point.
>>> 
>>> IOW, for hugetlbfs I don't know why we need to populate the pages at all if we
>>> simply want to know "whether we do still have enough space"..  And IIUC 2)
>>> above is the major issue you'd like to solve too.
>> 
>> To avoid page faults at runtime on access I think. Reservation <=
>> Preallocation.
> 
> Yes.  Besides my above question regarding max-length of virtio-mem device: we
> care most about private mappings of hugetlbfs/shmem here, am I right?
> 
> I'm thinking why we'd need MAP_PRIVATE of these at all for VM context.

One reason is that MAP_SHARED does not support mbind() - which should include hugetlbfs. I did not investigate other side effects / performance considerations on allocation.

Similarly, fallocate() does not respect/care about NUMA.

(And yes, NUMA for virtio-mem will be important).