Re: updatedb
From: Rene Herman
Date: Fri Jul 27 2007 - 07:12:02 EST
On 07/27/2007 11:26 AM, Mike Galbraith wrote:
On Fri, 2007-07-27 at 10:28 +0200, Rene Herman wrote:
I still wonder what the "the swap thing" is though. People just kept
saying that swap-prefetch helped which would seem to indicate their
problem didnt have anything to do with updatedb.
I haven't rummaged around in the VM in quite a long while, so don't know
exactly where the balance lies any more, and have never looked at
swap-prefetch, but the mechanism of how swap-prefetch can help the
"morning after syndrome" seems simple enough:
As far as I've googled things together, the below scenario won't happen:
Reclaim (swapout) a slew of application pages because there are
truckloads of utterly bored pages laying about when updatedb comes along
and introduces memory pressure in the middle of the night.
Ack (*)
Updatedb finishes, freeing some ram (doesn't matter how much)
Will be very little and swap-prefetch at least in its current form needs
more than very little to start doing anything:
http://ck.kolivas.org/patches/swap-prefetch/2.6.21-swap_prefetch-38.patch
| /*
| * Set max number of entries to 2/3 the size of physical ram as we
| * only ever prefetch to consume 2/3 of the ram.
| */
However, okay, let's just ignore that and pretend it kicks in even with the
little free memory updatedb itself left behind when it finished:
swap-prefetch detects idle CPU, and begins faulting swapped out pages
back in. In the process of doing so, memory pressure is generated, and
now these freshly accessed pages are a less lovely target than the now
aging VFS caches that updatedb bloated up, so they shrink back down
enough that the balance you had before updatedb ran is restored...
The story now again breaks down here. Over at:
http://lkml.org/lkml/2007/2/9/112
we have swap-prefetch's author saying:
| swap prefetch stores the data at the tail end of the lru list which
| means that even if you do want to use that ram for something else,
| the prefetched pages will be immediately dropped.
So those aging VFS caches will never be replaced by anything prefetched it
seems and any prefetching stops after the couple of pages that updatedb
itself freed have been taken again.
A subsequent bit in that same message reads:
| It can't help the updatedb scenario. Updatedb leaves the ram full and
| swap prefetch wants to cost as little as possible so it will never
| move anything out of ram in preference for the pages it wants to swap
| back in.
which I'll take as confirmation.
with the notable exception that cached data is now toast, so what you
gained by faulting god knows how frequently used pages back in isn't
_necessarily_ going to help you.
Also file-backed pages (such as the program binaries itself). However, _if_
prefetching were to take place, I'd be okay assuming it helps.
Heck, it could even step on what was left of your cached working set
after updatedb finished.
Given swap-prefetch's focus on being as free as possible I don't believe it
should hurt any. And yes, it's always going to help _some_ workload shifts
and as far as I'm concerned is a "makes sense" kind of tweak even if it's
not the be all end all so again, not against swap-prefetch or its merger or
anything...
Also, I know shit about the VFS so this may well be not very educated but to
me something like FADV_NOREUSE on a dirfd sounds like a much more promising
approach than the convoluted userspace schemes being discussed, if only
because it'll actually be implemented/used.
I like Andrew's mention of a future option... put that sucker and
everybody who looks like him in a resource limited container.
As to the (*) above -- now that I actually know about the "swappiness"
thing, shouldn't setting that to 0 around updatedb runs really be quite
effective and if it's not, is there anything to be fixed there? Bjoern
Steinbrink (added to CC) earlier in this thread tried that and said stuff
went weird.
Rene.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/