Re: [RFC PATCH 0/4] (Take 2): transcendent memory ("tmem") for Linux

From: Avi Kivity
Date: Sun Jul 12 2009 - 13:28:29 EST


On 07/12/2009 07:28 PM, Dan Magenheimer wrote:
Having no struct pages is also a downside; for example this
guest cannot
have more than 1GB of anonymous memory without swapping like mad.
Swapping to tmem is fast but still a lot slower than having
the memory
available.

Yes, true. Tmem offers little additional advantage for workloads
that have a huge variation in working set size that is primarily
anonymous memory. That larger scale "memory shaping" is left to
ballooning and hotplug.

And this is where the policy problems erupt. When do you balloon in favor of tmem? which guest do you balloon? do you leave it to the administrator? there's the host's administrator and the guests' administrators.

CMM2 solves this neatly by providing information to the host. The host can pick the least recently used page (or a better algorithm) and evict it using information from the guest, either dropping it or swapping it. It also provides information back to the guest when it references an evicted page: either the guest needs to recreate the page or it just needs to wait.

tmem makes life a lot easier to the hypervisor and to the guest, but
also gives up a lot of flexibility. There's a difference
between memory
and a very fast synchronous backing store.

I don't see that it gives up that flexibility. System adminstrators
are still free to size their guests properly. Tmem's contribution
is in environments that are highly dynamic, where the only
alternative is really sizing memory maximally (and thus wasting
it for the vast majority of time in which the working set is smaller).

I meant that once a page is converted to tmem, there's a limited amount of things you can do with it compared to normal memory. For example tmem won't help with a dcache intensive workload.

I'm certainly open to identifying compromises and layer modifications
that help meet the needs of both Xen and KVM (and others). For
example, if we can determine that the basic hook placement for
precache/preswap (or even just precache for KVM) can be built
on different underlying layers, that would be great!

I'm not sure preswap/precache by itself justifies tmem since it can be emulated by backing the disk with a cached file. What I'm missing in tmem is the ability for the hypervisor to take a global view on memory; instead it's forced to look at memory and tmem separately. That's fine for Xen since it can't really make any decisions on normal memory (lacking swap); on the other hand kvm doesn't map well to tmem since "free memory" is already used by the host pagecache.

--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/