Re: [Lse-tech] Re: [PATCH] [0/6] HUGETLB memory commitment

From: Ray Bryant
Date: Sun Mar 28 2004 - 16:28:55 EST

Martin J. Bligh wrote:
As I understood this originally, the suggestion was to reserve hugetlb pages at mmap() or shm_get() time so that the user would get an -ENOMEM at that time if there aren't enough hugetlb pages to (eventually) satisfy the request, as per the notion that we shouldn't modify the user API due to going with allocate on fault instead of hugetlb_prefault().

Yup, but there were two parts to it:

1. Stop hugepages using the existing overcommit pool for small pages, which breaks small page allocations by prematurely the pool.
2. Give hugepages their own over-commit pool, instead of prefaulting.

Personally I think we need both (as you seem to), but (1) is probably
more urgent.

Just to review: even if we allocate hugetlb pages at fault rather than at mmap() time, hugetlb pages are created either at system boot time (kernel parameter "hugepages=") or by setting /proc/sys/vm/nr_hugepages (or by using the corresponding sysctl). Once the set of hugepages is created this way, it never is changed by the act of allocating a huge page to a process. (Changing nr_pages can cause the number of unallocated hugetlbpages to be increased or decreased.)

The reason for pointing this out (apologies if this was obvious to all) is to emphaisze that hugetlbpages are not created at hugetlbpage allocation time (which is now done at mmap() time and we'd like to change it to happen at fault time).

So to stop hugepages from using the small page overcommit pool, we just need code in set_hugetlb_mem_size() to reduce the number of hugetlbpages created by that code.

As for (2), I'm a little confused there, as later you appear to agree with me that overcomitting hugetlbpages is likely not useful. Is it possible that you meant that there should be a list of hugetlbpages from which all allocations are made? If so, that is the way the code has always worked; step 1 was to create the list of hugetlbpages, and step 2 was to allocate them.

(Once again, if this is obvious to all, I apologize and we can dump the last 4 paragraphs into the bit bucket with no known effect on entropy in this universe, at least.)

Since the reservation belongs to the mapped object (file or segment), I've been storing the current file/segments's reservation in the file system dependent part of the inode. That way, it is easily accessible when the hugetlbfs file or SysV segment is removed and we can reduce the total number of reserved pages by that file's reservation at that time. This also allows us to handle the reservation in the absence of a vma, as per Andy'c comment below.

Do we need to store it there, or is one central pool number sufficient?
I would have thought it was ...

Yes, there is a central pool number indicating how many hugepages are reserved. The question is, when (and how) do you release that reservation? My take is that the reservation is associated with the file (for mmap) or segment for SysV.

For example, program A mmap()'s a hugetlbfs file, but only touches part of the pages. Program B then mmap()'s the same file with the same size, etc. When program B does the mmap() the previous reservation should still be in place, right? (The file is persistent in the page cache even if it does not persist over reboot, so the 2nd program is exepecting to see the data that the first program put there.)

Ditto for a SysV segement.

So one can't release the reservation when the current process doing the mmap() goes away, one has to release the reservation when the file/segment is deleted. Since both mmap() and shmget() create an inode, and the inode is released by hugetlbfs_drop_inode() and friends, it seemed simplest to put the size of the mapped object's reservation in the inode.

The global count of reserved pages (the "central pool number" in your note), is incremented at mmap() time (well, actually done by hugetlbfs_file_mmap() for both mmap() and shmget()) and decremented at hugetlbfs_drop_inode() time. If at mmap() time, incrementing the global reservation count would make the global reserved pages count > the number of hugetlbpages, we fail the mmap() with -ENONMEM.

At least that is the way my 2.4.21 code works. Does that make things clearer?

Admittedly this doesn't alow one to request that hugetlbpages be overcommitted, or to handle problems caused to the "normal" page overcommit code due to the presence of hugepages. But we figure that anyone that is actually using hugetlb pages is likely to take over almost all of main memory anyway in a single job, so overcommit doesn't make much sense to us.

Seeing as you can't swap them, overcommitting makes no sense to me
either ;-)


This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
Lse-tech mailing list

Best Regards,
Ray Bryant
512-453-9679 (work) 512-507-7807 (cell)
raybry@xxxxxxx raybry@xxxxxxxxxxxxx
The box said: "Requires Windows 98 or better",
so I installed Linux.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at