Re: smpnice loadbalancing with high priority tasks

[Siddha, Suresh B]

On Sun, Apr 02, 2006 at 05:08:33PM +1000, Peter Williams wrote:
> Siddha, Suresh B wrote:
> > Peter,
> > 
> > There are still issues which we need to address.. These are surfacing
> > as we are patching issue by issue(instead of addressing the root issue, which
> > is: presence of high priority tasks messes up load balancing of normal
> > priority tasks..)
> > 
> > for example
> > 
> > a) on a simple 4-way MP system, if we have one high priority and 4 normal
> > priority tasks, with smpnice we would like to see the high priority task
> > scheduled on one cpu, two other cpus getting one normal task each and the
> > fourth cpu getting the remaining two normal tasks. but with smpnice that 
> > extra normal priority task keeps jumping from one cpu to another cpu having
> > the normal priority task.
> > 
> > This is because of the busiest_has_loaded_cpus, nr_loaded_cpus logic.. We
> > are not including the cpu with high priority task in max_load calculations
> > but including that in total and avg_load calcuations.. leading to max_load <
> > avg_load and load balance between cpus running normal priority tasks(2 Vs 1)
> > will always show imbalanace as one normal priority and the extra normal
> > priority task will keep moving from one cpu to another cpu having
> > normal priority task..
> 
> I can't see anything like this in the code.  

Don't you see a condition where max_load < avg_load(as mentioned in the
above example) and in this case, code ignores avg_load and imbalance
will aways be the extra normal priority task( coming from
"max_load - busiest_load_per_task") and this normal priority task keeps 
hopping from one cpu to another cpu having normal priority task..

> Can you send a patch to fix 
> what you think the problem in the is?

I am looking at ways in fixing all these issues cleanly... I don't have
a clean solution yet...

> 
> The effect you describe can be caused by other tasks running on the 
> system (see below for fuller description).
> 
> > 
> > b) on a simple DP system, if we have two high priority and two normal priority
> > tasks, ideally we should schedule one high and one normal priority task on 
> > each cpu.. current code doesn't find an imbalance if both the normal priority
> > tasks gets scheduled on the same cpu(running one high priority task)

if the system goes into this state(whatever the reason may be...), how can
the current code detect this and resolve this imbalance... (by reviewing the
code, it shows that imbalance in this particular situation can't be detected...)


> This is one of my standard tests and it works for me.  The only time the 

Force your test into the above scenario and see how it behaves... if you
simply start new processes, then exec balance will nicely balance...
things go bad only when we go into the described above state..

I can give numerous other examples which fails

c) DP system: if the cpu-0 has two high priority and cpu-1 has one normal
priority task, how can the current code detect this imbalance..

d) 4-way MP system: if the cpu-0 has two high priority tasks, cpu-1 has
one high priority and four normal priority and cpu-2,3 each has one
high priority task.. how does the current code distribute the normal
priority tasks among cpu-1,2,3... (in this case, max_load will always
point to cpu-0 and will never distribute the noraml priority tasks...)

by giving these examples I am just pointing out various corner conditions..

> This (in my opinion) is what you are seeing.

Simple review of the current code is showing all these problems... when you
start new tasks, all will be well because of fork/exec balance... but
during run time, if the system goes to a particular state, current imbalance
code behaves poorly in the presence of high priority tasks...

thanks,
suresh
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/