Re: 2.6.12 Performance problems

[Helge Hafting]

Danial Thom wrote:

> --- Jesper Juhl <jesper.juhl@xxxxxxxxx> wrote:
>
>  
>
> > On 8/21/05, Danial Thom <danial_thom@xxxxxxxxx>
> > wrote:
> >    
> >
> > > I just started fiddling with 2.6.12, and
> > >      
> > there
> >    
> >
> > > seems to be a big drop-off in performance
> > >      
> > from
> >    
> >
> > > 2.4.x in terms of networking on a
> > >      
> > uniprocessor
> >    
> >
> > > system. Just bridging packets through the
> > > machine, 2.6.12 starts dropping packets at
> > > ~100Kpps, whereas 2.4.x doesn't start
> > >      
> > dropping
> >    
> >
> > > until over 350Kpps on the same hardware
> > >      
> > (2.0Ghz
> >    
> >
> > > Opteron with e1000 driver). This is pitiful
> > > prformance for this hardware. I've
> > > increased the rx ring in the e1000 driver to
> > >      
> > 512
> >    
> >
> > > with little change (interrupt moderation is
> > >      
> > set
> >    
> >
> > > to 8000 Ints/second). Has "tuning" for MP
> > > destroyed UP performance altogether, or is
> > >      
> > there
> >    
> >
> > > some tuning parameter that could make a
> > >      
> > 4-fold
> >    
> >
> > > difference? All debugging is off and there
> > >      
> > are
> >    
> >
> > > no messages on the console or in the error
> > >      
> > logs.
> >    
> >
> > > The kernel is the standard kernel.org dowload
> > > config with SMP turned off and the intel
> > >      
> > ethernet
> >    
> >
> > > card drivers as modules without any other
> > > changes, which is exactly the config for my
> > >      
> > 2.4
> >    
> >
> > > kernels.
> > >
> > >      
> > If you have preemtion enabled you could disable
> > it. Low latency comes
> > at the cost of decreased throughput - can't
> > have both. Also try using
> > a HZ of 100 if you are currently using 1000,
> > that should also improve
> > throughput a little at the cost of slightly
> > higher latencies.
> >
> > I doubt that it'll do any huge difference, but
> > if it does, then that
> > would probably be valuable info.
> >
> >    
> Ok, well you'll have to explain this one:
>
> "Low latency comes at the cost of decreased
> throughput - can't have both"
>  
Configuring "preempt" gives lower latency, because then
almost anything can be interrupted (preempted).  You can then
get very quick responses to some things, i.e. interrupts and such.

The cost comes, because _something_ was interrupted, something
that instead would run to completion first in a kernel made without 
"preempt".
So that other thing, whatever it is, got slower. 

And the problem is bigger than merely "things happens in a different order".
Switching the cpu from one job to another have a big overhead.  
Particularly,
the cpu caches have to be refilled more often, which takes time.  
Running one
big job to completion fills the cache with that job's data _once_.  If 
the job
is preempted a couple of times you have to bring it into cache three
times instead, and that will cost you, performance wise.

This is not _necessarily_ your problem, but trying a 2.6 kernel without 
preempt
and with hz=100 (both things configurable through normal kernel 
configuration)
will clearly show if this is the problem in your case.  If you're lucky, 
this is all
you need to get your performance back.  If not, then at least it is an
important datapoint for those trying to figure it out.  Nobody here want
2.6 to have 1/4 of the performance of 2.4!

Helge Hafting
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/