Re: [PATCH 0/8] MADV_FREE support

From: David Rientjes
Date: Wed Nov 04 2015 - 15:19:28 EST

On Sun, 1 Nov 2015, Daniel Micay wrote:

> It can definitely be improved to cooperate well with THP too. I've been
> following the progress, and most of the problems seem to have been with
> the THP and that's a very active area of development. Seems best to deal
> with that after a simple, working implementation lands.
> The best aspect of MADV_FREE is that it completely avoids page faults
> when there's no memory pressure. Making use of the freed memory only
> triggers page faults if the pages had to be dropped because the system
> ran out of memory. It also avoids needing to zero the pages. The memory
> can also still be freed at any time if there's memory pressure again
> even if it's handed out as an allocation until it's actually touched.
> The call to madvise still has significant overhead, but it's much
> cheaper than MADV_DONTNEED. Allocators will be able to lean on the
> kernel to make good decisions rather than implementing lazy freeing
> entirely on their own. It should improve performance *and* behavior
> under memory pressure since allocators can be more aggressive with it
> A nice future improvement would be landing MADV_FREE_UNDO feature to
> allow an attempt to pin the pages in memory again. It would make this
> work very well for implementing caches that are dropped under memory
> pressure. Windows has this via MEM_RESET (essentially MADV_FREE) and
> MEM_RESET_UNDO. Android has it for ashmem too (pinning/unpinning). I
> think browser vendors would be very interested in it.

This sounds similar to what I was proposing to prevent thp splits when
there is no memory pressure.

MADV_SPLITTABLE marks ranges of memory as free and the underlying thp may
be split if there is no memory pressure. Under memory pressure, it acts
identical to MADV_DONTNEED. Without memory pressure, the range is
enqueued on an lru for the memcg that the vma's mm owner belongs to
(global for !CONFIG_MEMCG). It is also linked on a per-vma list for the
range. Anytime the vma is manipulated, the MADV_SPLITTABLE ranges are
also fixed up.

On subsequent memory pressure, the memcg hierarchy lru list is iterated
(global for !CONFIG_MEMCG) and the MADV_SPLITTABLE ranges are actually
zapped (including thp split if necessary) and the memory is really freed
to the system.

MADV_UNSPLITTABLE marks ranges of memory that have already been freed
through MADV_SPLITTABLE as being used again. If there was no memory
pressure and the MADV_SPLITTABLE was simply enqueued on the lru list, it
is removed from that list after the range has been zeroed with the same
user-facing semantics as MADV_DONTNEED. Otherwise, nothing is done since
the ptes are already zapped and we'll incur a refault.

The change to tcmalloc is simple: use MADV_SPLITTABLE instead of
MADV_DONTNEED when freeing memory to the system and use MADV_UNSPLITTABLE
when returning memory that has been already freed to the system.

This works well in experimentation when 100% of heap backed by thp with no
memory pressure. This is a type of lazy-free that prevents thp memory
from being split without memory pressure.

I was wondering if this could become part of MADV_FREE behavior with the
MADV_FREE_UNDO behavior as the equivalent to my MADV_UNSPLITTABLE. If
there is no ground to be shared, mine is just implemented seperately, but
I'm trying to avoid additional system calls required for malloc
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at