Re: computing drop-able caches

[Johannes Weiner]

On Fri, Feb 12, 2016 at 01:46:54PM -0800, Dave Hansen wrote:
> On 02/12/2016 10:01 AM, Khalid Mughal (khalidm) wrote:
> > If you look at the attached pdf, you will notice that OOM messages start
> > to appear when memAvailable is showing 253MB (259228 KB) Free, memFree is
> > 13.5MB (14008 KB) Free, and dropcache based calculation ³Available memory²
> > is showing 21MB (21720 KB) Free.
> > 
> > So, it appears that memAvailable is not as accurate, especially if data is
> > used to warn user about system running low on memory.
> 
> Yep, that's true.
> 
> But, MemAvailable is calculated from some very cheap counters.  The
> "dropcache-based-calculation" requires iterating over every 4k page
> cache page in the system.

It's also completely off when the dominating cache consumer uses
mmap() instead of buffered IO.

> We track dirty and writebackw with counters, so we should theoretically
> be able to pull those out of MemAvailable fairly cheaply.

Yeah, that sounds like a good idea.

But the fact remains that this will never be fully accurate, and there
will always be scenarios where the system will thrash and trigger OOM
before this counter depletes, simply because maintaining uptodate heat
information of the page cache would be crazy expensive.

IMO, the best way to use the MemAvailable counter is to calculate a
utilization percentage against MemTotal, and then maintaining a
healthy number like 80-90% - depending on expected runtime variance
and an educated guess of how hot the page cache is.