[RFC] Another Para-Virtualization page recycler. Empty Guest OS free pages every few seconds

[XaviLi]

PPR (Per Page Recycler) is a para virtualization driver currently available for KVM hosts and Linux/Windows guests. With PPR, every page freed to Guest OS can be recycled in seconds by hypervisor. Therefore, VMs can dynamical allocate/free pages from hypervisor according to applicationâs request. This issue can result in a memory density boost in many cases with little CPU cost. PPR is a component of an open source solution named Dynamic VM:
https://github.com/baibantech/dynamic_vm.git

patch can be found here: https://github.com/baibantech/dynamic_vm/tree/master/dynamic_vm_0.5

The guest side is very simple. PPR have a hook at the entry of page-free interface of Guest OS to mark every freed pages. Marked pages can be recycled and replaced by read-only zero page in one scan period. If the page was allocated again before the recycling happens, the mark was simply canceled by PPRâs page-allocation interface hook.

The guest driver works in a per-page way other than VirtIO balloonâs batch mode. The recycling issue is based on a lockless operation. The guest and host does not send any interrupt to each other for this.

We ran a test on our machine. Host use about 6500+ CPU cycles averagely for recycling each page. In which 3900 cycles is cost by the Linux kernel page-table operation. The guest hook cost about 300 cycles for mark or unmark each page. It is cheap enough in many cases. The daemon cost 5% of one CPU when the scanning found nothing to recycle. Other threads only move when there is work to do.

Saving memory often leads to a CPU-memory tradeoff topic. Here thing is different. In memory intensive usage we often see page-deduplication applied. It is very much cheaper to recycle a page than dedup it. In that case we can use PPR to win CPU and memory both.

We have a test to demonstrate how density was boosted. In which PPR works with a multithread page deduplication engine named PageONE rather than KSM. PageONE is going to be discussed in another topic.

Test environment
DELL R730
Intel Xeon E5-2650 v4 (2.20 GHz, of Cores 12, threads 24); 
256GB RAM
Host OS: Ubuntu server 14.04 Host kernel: 4.4.1
Qemu: 2.9.0
Guest OS: Ubuntu server 16.04 Guest kernel: 4.4.76

Test VMsâ memory size was configured to be 4GB. Each VM ran an application allocate/free memory every 30 minutes. The top memory allocation size of each period is 3GB and average size is 150MB. We ran many such VMs in the same time same machine to see the total host-side memory cost.


Raw KVM
Result: 67 VMs used 230GB memory. 

KVM + KSM 
Configuration: sleep_millisecs = 0, pages_to_scan = 1000000
Result: 73 VMs used 230GB memory. KSM daemon cost 100% of one CPU

KVM + PPR + PageONE:
Configuration: 
work threads =12 reclaim_period = 10 secs merge_period = 600 secs 
(Which means freed pages was recycled each 10 secs, pages were merged if not modified in 2 merge_periods)
Result: 516 VMs run together. The total used memory goes up and down between 218GB and 139GB, averagely 182GB. 

PPR and PageONE shared a daemon thread and 12 work threads. We test their CPU cost together because the two features are sharing threads while offloading for each other. 

The average CPU cost of work threads is 6.1%. Daemon threadâs cost is 22.1%

Here we tried other combinations in the same test case:
reclaim_period(sec) merge_period(sec) average memory(GB) work threads' average CPU daemon's CPU
5 200 164 6.8% 34%
5 600 176 6.3% 35%
10 200 171 6.6% 22%
10 600 182 6.1% 22%
NOTE: It is a high pressure test. PPR was recycling 3.2TB memory per hour. In normal case PPR costs far less CPU than that.