Re: miserable performance of 2.6.21 under network load

From: Chris Snook
Date: Tue Jul 24 2007 - 15:23:18 EST


Aaron Porter wrote:
On Tue, Jul 24, 2007 at 08:48:00PM +0200, Andi Kleen wrote:
Aaron Porter <atporter@xxxxxxxxxxx> writes:

I'm in the process up upgrading a pool of apache servers from
2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
Under identical network load, 2.6.21 has a load average more than 3 times
higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
and our network traffic drops off rapidly. The end result is that 2.6.17
performs twice as well under this load.
Can you oprofile it?

# opreport -l
CPU: AMD64 processors, speed 1994.52 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
samples % app name symbol name
914379 48.8404 vmlinux-2.6.21.5 check_poison_obj
341920 18.2632 vmlinux-2.6.21.5 poison_obj

I bet you have CONFIG_DEBUG_SLAB turned off in your 2.6.17 kernel, and turned on in your 2.6.21 kernel.

-- Chris
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/