HPL Benchmark performance degradation of kernel 2.6.24.3 vs 2.6.23.14

[Allan Menezes]

Hi,
   I have a five node intel Q6600 quad core cluster and I benchmarked 
it with open source open mpi software using fc8 and it's supplied 
kernels recompiled and that of kernel.org with kernel 2.6.23.14 and 
2.6.24.3.
With GotoBlas v 1.24 and open mpi beta both cases (v 1.3a) for kernels 
2.6.23.14 with web100 i get 158GFlops.
But when i recompile with web100 for kernel 2.6.24 / without web100 and 
having 6gig DDR2800MHz ram on each node i get only 28GFLOPS AND 
22GFLOPS  for 5 nodes whereas with or without web 100 for kernel 
2.6.23.14 i get 156-8 GfLOPS. wITH OR WITHOUT web 100 i get for kernel 
2.6.24.3 22- 28 Gflops for 5 nodes.!
Why is there a performance drop in kernel 2.6.24.3 All else hardware is 
the same!
For inter node communication i use three pci express gig eth cards ( 2 
intel and one syskonnect ) per node and using nptcp of netpipe their 
performance of intel and syskonnect cards in both kernels measured point 
to point is 880MBPS approx for all three cards with measured using 
netpipe for tcp with kernel 2.6.24.3 and 2.6.23.14 . I am also using 
three switches gigabit with high bisection b/w for these eth cards 
(copper) with 3 different subnets
Yet I am getting a substantial performance drop keeping the hardware and 
openmpi and hpl and gotoblas same. Can some one help me figure out why?
Please find attached my kernel's .config
Cheers,
Allan Menezes

Attachment: .config
Description: application/xml