Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plansfor 2.6.23]

[david]

On Sun, 29 Jul 2007, Rene Herman wrote:

> On 07/28/2007 11:00 PM, david@xxxxxxx wrote:
>
> > >  many -mm users use it anyway? He himself said he's not convinced of 
> > >  usefulness having not seen it help for him (and notice that most 
> > >  developers are also users), turned it off due to it annoying him at some 
> > >  point and hasn't seen a serious investigation into potential downsides.
> >
> >  if that was the case then people should be responding to the request to
> >  get it merged with 'but it caused problems for me when I tried it'
> >
> >  I haven't seen any comments like that.
>
> So you're saying Andrew did not say that? You're jumping to the conclusion 
> that I am saying that it's causing problems.

I don't remember anyone saying that it actually caused problems (including 
both you and andrew). I (and others) have been trying to learn what 
problems people believe it has in the hope that they can be addressed one 
way or another.

> > > >   that the only significant con left is the potential to mask other
> > > >   problems.
> > > 
> > >  Which is not a madeup issue, mind you. As an example, I just now tried 
> > >  GNU locate and saw it's a complete pig and specifically unsuitable for 
> > >  the low memory boxes under discussion. Upon completion, it actually 
> > >  frees enough memory that swap-prefetch _could_ help on some boxes, while 
> > >  the real issue is that they should first and foremost dump GNU locate.
> >
> >  I see the conclusion as being exactly the opposite.
>
> And now you do it again :-) There is no conclusion -- just the inescapable 
> observation that swap-prefetch was (or may have been) masking the problem of 
> GNU locate being a program that noone in their right mind should be using.

isn't your conclusion then that if people just stopped useing that version 
of updatedb the problem would be solved and there would be no need for the 
swap prefetch patch? that seemed to be what you were strongly implying (if 
not saying outright)

> >  so there is a legitimate situation where swap-prefetch will help
> >  significantly, what is the downside that prevents it from being included?
>
> People being unconvinced it helps all that much, no serious investigation 
> into possible downsides and no consideration of alternatives is three I've 
> personally heard.
>
> You don't want to merge a conceptually core VM feature if you're not really 
> convinced. It's not a part of the kernel you can throw a feature into like 
> you could some driver saying "ah, heck, if it makes someone happy" since 
> everything in the VM ends up interacting -- that in fact is actually the hard 
> part of VM as far as I've seen it.
>
> And in this situation the proposed feature is something that "papers over a 
> problem" by design -- where it could certainly be that the problem is not 
> solveable in another way simply due to the kernel not growing the possiblity 
> to read user's minds anytime soon (which some might even like to rephrase as 
> "due to no problem existing") but that this gets people a bit anxious is not 
> surprising.

people who have lots of memory and so don't use swap will never see the 
benifit of this patch. over the years many people have investigated the 
problem and tried to address it in other ways (the better version of 
updatedb is an attempt to fix it for that program as an example), but 
there is still a problem.

I agree that tinkering with the core VM code should not be done lightly, 
but this has been put through the proper process and is stalled with no 
hints on how to move forward.

> >  I've seen it mentioned that there is still a maintainer but I missed who
> >  it is, but I haven't seen any concerns that can be addressed, they all
> >  seem to be 'this is a core concept, people need to think about it' or 'but
> >  someone may find a better answer in the future' type of things. it's
> >  impossible to address these concerns directly.
>
> So do it indirectly. But please don't just say "it help some people (not me 
> mind you!) so merge it and if you don't it's all just politics and we can't 
> do anything about it anyway". Because that's mostly what I've been hearing.
>
> And no, I'm not subscribed to any ck mailinglists nor do I hang around its 
> IRC community which will can account for part of that. I expect though that 
> the same holds for the people that actually matter in this, such as Andrew 
> Morton and Nick Piggin.
>
> -- 1: people being unconvinced it helps all that much
>
> At least partly caused by the updatedb i/dcache red herring that infected 
> this issue. Also, at the point VM  pressure has mounted high enough to cause 
> enough to be swapped out to give you a bad experience, a lot of other things 
> have been dropped already as well.
>
> It's unsurprising though that it would for example help the issue of 
> openoffice with a large open spreadsheet having been thrown out overnight 
> meaning it's a matter of deciding whether or not this is an important enough 
> issue to fix inside the VM with something like swap-prefetch.
>
> Personally -- no opinion, I do not experience the problem (I even switch off 
> the machine at night and do not run cron at all).

forget the nightly cron jobs for the moment. think of this scenerio. you 
have your memory fairly full with apps that you have open (including 
firefox with many tabs), you receive a spreadsheet you need to look at, so 
you fire up openoffice to look at it. then you exit openoffice and try to 
go back to firefox (after a pause while you walk to the printer to get 
the printout of the spreadsheet), only to find that it's going to be 
sluggish becouse it got swapped out due to the preasure from openoffice.

no nightly cron job needed, just enough of a memory hog or a small enough 
amount of ram to have your working set exceed it.

> -- 2: no serious investigation into possible downsides
>
> Swap-prefetch tries hard to be as free as possible and it seems to largely be 
> succeeding at that. Thing that (obviously -- as in I wouldn't want to state 
> it's the only possible worry anyone could have left) remains is the "papering 
> over effect" it has by design that one might not care for.
>
> -- 3: no serious consideration of possible alternatives
>
> Tweaking existing use-oce logic is one I've heard but if we consider the 
> i/dcache issue dead, I believe that one is as well. Going to userspace is 
> another one. Largest theoretical potential. I myself am extremely sceptical 
> about the Linux userland, and largely equate it with "smallest _practical_ 
> potential" -- but that might just be me.
>
> A larger swap granularity, possible even a self-training granularity. Up to 
> now, seeks only get costlier and costlier with respect to reads with every 
> generation of disk (flash would largely overcome it though) and doing more in 
> one read/write _greatly_ improves throughput, maybe up to the point that 
> swap-prefetch is no longer very useful. I myself don't know about the 
> tradeoffs involved.

larger swap granularity may help, but waiting for the user to need the ram 
and have to wait for it to be read back in is always going to be worse for 
the user then pre-populating the free memory (for the case where the 
pre-population is right, for other cases it's the same). so I see this as 
a red herring

> Any other alternatives?
>
> Any 4th and higher points?

there are fully legitimate situations where this is useful, the 'papering 
over' effect is not referring to these, it's referring to other possible 
problems in the future. I see this argument as being in the same catagory 
as people wanting to remove the old, working driver for some hardware in 
favor of the new, unreliable driver for the same hardware in order to get 
more bug reports to find the holes in the new driver. that's causing users 
unessasary pain and within the last week Linus was takeing a driver author 
to task for attempting exactly that IIRC (and yes, there does come a point 
where there are no further bugs known in the new driver, and it appears to 
do everything the old driver did that you do remove the old driver, but 
you don't remove it early to help the new driver stabilize)

David Lang

> Rene.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/