Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plansfor 2.6.23]

From: Rene Herman
Date: Sun Jul 29 2007 - 10:04:33 EST


On 07/29/2007 01:41 PM, david@xxxxxxx wrote:

And now you do it again :-) There is no conclusion -- just the inescapable observation that swap-prefetch was (or may have been) masking the problem of GNU locate being a program that noone in their right mind should be using.

isn't your conclusion then that if people just stopped useing that version of updatedb the problem would be solved and there would be no need for the swap prefetch patch? that seemed to be what you were strongly implying (if not saying outright)

No. What I said outright, every single time, is that swap-prefetch in itself seems to make sense. And specifically that even if the _direct_ problem is a crummy program, it _still_ makes sense generally. Every single time.

But see -- you failed to notice this because you guys are stuck in this dumb adversary "us against them" thing so inherent of (online) communities, where you sit around your own habitats patting each other on the back for extended periods of time and then every once a while go out clinging on to each other vigorously and going "boo! hiss!" at the big bad outside world.

I already got overly violent at one point in this thread so I'll leave out any further references to sense-deprived fanboy-culture but please, I said every single time that I'm not against swap-prefetch. I cannot communicate when I'm not being read.

I agree that tinkering with the core VM code should not be done lightly, but this has been put through the proper process and is stalled with no hints on how to move forward.

It has not. Concerns that were raised (by specifically Nick Piggin) weren't being addressed.

forget the nightly cron jobs for the moment. think of this scenerio. you have your memory fairly full with apps that you have open (including firefox with many tabs), you receive a spreadsheet you need to look at, so you fire up openoffice to look at it. then you exit openoffice and try
to go back to firefox (after a pause while you walk to the printer to
get the printout of the spreadsheet)

And swinging a dead rat from its tail facing east-wards while reciting Documentation/CodingStyle.

Okay, very very sorry, that was particularly childish, but that "walking to the printer" is ofcourse completely constructed and this _is_ something to take into account. Swap-prefetch wants to be free, which (also again) it is doing a good job at it seems, but this also means that it waits for the VM to be _very_ idle before it does anything and as such, we cannot just forget the "nightly" scenario and pretend it's about something else entirely. As long as the machine's being used, swap-prefetch doesn't kick in.

Which is a good feature for swap-prefetch, but also something that needs to weighed alongside its other features in a discussion of alternatives, where for example something like a larger swap granularity would not have anything of the sort to take into account. If it were about walks to the printer, we could shelve the issue as being of too limited practical use for inclusion.

-- 2: no serious investigation into possible downsides

Swap-prefetch tries hard to be as free as possible and it seems to largely be succeeding at that. Thing that (obviously -- as in I wouldn't want to state it's the only possible worry anyone could have left) remains is the "papering over effect" it has by design that one might not care for.

Arjan van de Ven made another point here about seeking away due to swap-prefetch (just) before the next request comes in, but that's probably a bit of a non-issue in practice with the "very idle" precondition.

-- 3: no serious consideration of possible alternatives

Tweaking existing use-oce logic is one I've heard but if we consider the i/dcache issue dead, I believe that one is as well. Going to userspace is another one. Largest theoretical potential. I myself am extremely sceptical about the Linux userland, and largely equate it with "smallest _practical_ potential" -- but that might just be me.

A larger swap granularity, possible even a self-training granularity. Up to now, seeks only get costlier and costlier with respect to reads with every generation of disk (flash would largely overcome it though) and doing more in one read/write _greatly_ improves throughput, maybe up to the point that swap-prefetch is no longer very useful. I myself don't know about the tradeoffs involved.

larger swap granularity may help, but waiting for the user to need the ram and have to wait for it to be read back in is always going to be worse for the user then pre-populating the free memory (for the case where the pre-population is right, for other cases it's the same). so I see this as a red herring

I saw Chris Snook make a good post here and am going to defer this part to that discussion:

http://lkml.org/lkml/2007/7/27/421

But no, it's not a red herring if _practically_ speaking the swapin is fast enough once started that people don't actually mind anymore since in that case you could simply do without yet more additional VM complexity (and kernel daemon).

there are fully legitimate situations where this is useful, the 'papering
over' effect is not referring to these, it's referring to other possible
problems in the future.

No, it's not just future. Just look at the various things under discussion now such as improved use-once and better swapin.

Rene.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/