Re: [PATCH 0/3] Unmapped page cache control (v5)

[Balbir Singh]

* KOSAKI Motohiro <kosaki.motohiro@xxxxxxxxxxxxxx> [2011-03-31 14:40:33]:

> > 
> > The following series implements page cache control,
> > this is a split out version of patch 1 of version 3 of the
> > page cache optimization patches posted earlier at
> > Previous posting http://lwn.net/Articles/425851/ and analysis
> > at http://lwn.net/Articles/419713/
> > 
> > Detailed Description
> > ====================
> > This patch implements unmapped page cache control via preferred
> > page cache reclaim. The current patch hooks into kswapd and reclaims
> > page cache if the user has requested for unmapped page control.
> > This is useful in the following scenario
> > - In a virtualized environment with cache=writethrough, we see
> >   double caching - (one in the host and one in the guest). As
> >   we try to scale guests, cache usage across the system grows.
> >   The goal of this patch is to reclaim page cache when Linux is running
> >   as a guest and get the host to hold the page cache and manage it.
> >   There might be temporary duplication, but in the long run, memory
> >   in the guests would be used for mapped pages.
> >
> > - The option is controlled via a boot option and the administrator
> >   can selectively turn it on, on a need to use basis.
> > 
> > A lot of the code is borrowed from zone_reclaim_mode logic for
> > __zone_reclaim(). One might argue that the with ballooning and
> > KSM this feature is not very useful, but even with ballooning,
> > we need extra logic to balloon multiple VM machines and it is hard
> > to figure out the correct amount of memory to balloon. With these
> > patches applied, each guest has a sufficient amount of free memory
> > available, that can be easily seen and reclaimed by the balloon driver.
> > The additional memory in the guest can be reused for additional
> > applications or used to start additional guests/balance memory in
> > the host.
> 
> If anyone think this series works, They are just crazy. This patch reintroduce
> two old issues.
> 
> 1) zone reclaim doesn't work if the system has multiple node and the
>    workload is file cache oriented (eg file server, web server, mail server, et al). 
>    because zone recliam make some much free pages than zone->pages_min and
>    then new page cache request consume nearest node memory and then it
>    bring next zone reclaim. Then, memory utilization is reduced and
>    unnecessary LRU discard is increased dramatically.
> 
>    SGI folks added CPUSET specific solution in past. (cpuset.memory_spread_page)
>    But global recliam still have its issue. zone recliam is HPC workload specific 
>    feature and HPC folks has no motivation to don't use CPUSET.
>

I am afraid you misread the patches and the intent. The intent to
explictly enable control of unmapped pages and has nothing
specifically to do with multiple nodes at this point. The control is
system wide and carefully enabled by the administrator.
 
> 2) Before 2.6.27, VM has only one LRU and calc_reclaim_mapped() is used to
>    decide to filter out mapped pages. It made a lot of problems for DB servers
>    and large application servers. Because, if the system has a lot of mapped
>    pages, 1) LRU was churned and then reclaim algorithm become lotree one. 2)
>    reclaim latency become terribly slow and hangup detectors misdetect its
>    state and start to force reboot. That was big problem of RHEL5 based banking
>    system.
>    So, sc->may_unmap should be killed in future. Don't increase uses.
> 

Can you remove sc->may_unmap without removing zone_reclaim()? The LRU
churn can be addressed at the time of isolation, I'll send out an
incremental patch for that.

> 
> But, I agree that now we have to concern slightly large VM change parhaps
> (or parhaps not). Ok, it's good opportunity to fill out some thing.
> Historically, Linux MM has "free memory are waste memory" policy, and It
> worked completely fine. But now we have a few exceptions.
> 
> 1) RT, embedded and finance systems. They really hope to avoid reclaim
>    latency (ie avoid foreground reclaim completely) and they can accept 
>    to make slightly much free pages before memory shortage.
> 
> 2) VM guest
>    VM host and VM guest naturally makes two level page cache model. and
>    Linux page cache + two level don't work fine. It has two issues
>    1) hard to visualize real memory consumption. That makes harder to 
>       works baloon fine. And google want to visualize memory utilization
>       to pack in more jobs.
>    2) hard to make in kernel memory utilization improvement mechanism.
> 
> 
> And, now we have four proposal of utilization related issues.
> 
> 1) cleancache (from Oracle)

Cleancache requires both hypervisor and guest support. With these
patches, Linux can run well under hypverisor if we know the hypversior
does a lot of the IO and maintains the cache.

> 2) VirtFS (from IBM)
> 3) kstaled (from Google)
> 4) unmapped page reclaim (from you)
> 
> Probably, we can't merge all of them and we need to consolidate some 
> requirement and implementations.
> 
> 
> cleancache seems most straight forward two level cache handling for
> virtalization. but it has soem xen specific mess and, currently, don't fit RT
> usage. VirtFS has another interesting de-duplication idea. But filesystem based

I am planning to work on deduplication at KSM level at some point, but
I think again you misread the intention of these patches.

> implemenation naturally inherit some vfs interface limitations.
> Google approach is more unique. memcg don't have double cache
> issue, therefore they only want to visualize it.
> 
> Personally I think cleancache or other multi level page cache framework
> looks promising. but another solution is also acceptable. Anyway, I hope 
> to everyone back 1000feet bird eye at once and sorting out all requiremnt 
> with all related person. 
>

Thanks for the review! 

-- 
	Three Cheers,
	Balbir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/