Re: Where is the performance bottleneck?

[Holger Kiehl]

On Mon, 29 Aug 2005, Vojtech Pavlik wrote:

> On Mon, Aug 29, 2005 at 06:20:56PM +0000, Holger Kiehl wrote:
> > Hello
> >
> > I have a system with the following setup:
> >
> >     Board is Tyan S4882 with AMD 8131 Chipset
> >     4 Opterons 848 (2.2GHz)
> >     8 GB DDR400 Ram (2GB for each CPU)
> >     1 onboard Symbios Logic 53c1030 dual channel U320 controller
> >     2 SATA disks put together as a SW Raid1 for system, swap and spares
> >     8 SCSI U320 (15000 rpm) disks where 4 disks (sdc, sdd, sde, sdf)
> >       are on one channel and the other four (sdg, sdh, sdi, sdj) on
> >       the other channel.
> >
> > The U320 SCSI controller has a 64 bit PCI-X bus for itself, there is
> > no other device on that bus. Unfortunatly I was unable to determine at
> > what speed it is running, here the output from lspci -vv:
>
> > How does one determine the PCI-X bus speed?
>
> Usually only the card (in your case the Symbios SCSI controller) can
> tell. If it does, it'll be most likely in 'dmesg'.
There is nothing in dmesg:

   Fusion MPT base driver 3.01.20
   Copyright (c) 1999-2004 LSI Logic Corporation
   ACPI: PCI Interrupt 0000:02:04.0[A] -> GSI 24 (level, low) -> IRQ 217
   mptbase: Initiating ioc0 bringup
   ioc0: 53C1030: Capabilities={Initiator,Target}
   ACPI: PCI Interrupt 0000:02:04.1[B] -> GSI 25 (level, low) -> IRQ 225
   mptbase: Initiating ioc1 bringup
   ioc1: 53C1030: Capabilities={Initiator,Target}
   Fusion MPT SCSI Host driver 3.01.20

> > Anyway, I thought with this system I would get theoretically 640 MB/s using
> > both channels.
>
> You can never use the full theoretical bandwidth of the channel for
> data. A lot of overhead remains for other signalling. Similarly for PCI.
>
> > I tested several software raid setups to get the best possible write
> > speeds for this system. But testing shows that the absolute maximum I
> > can reach with software raid is only approx. 270 MB/s for writting.
> > Which is very disappointing.
>
> I'd expect somewhat better (in the 300-400 MB/s range), but this is not
> too bad.
>
> To find where the bottleneck is, I'd suggest trying without the
> filesystem at all, and just filling a large part of the block device
> using the 'dd' command.
>
> Also, trying without the RAID, and just running 4 (and 8) concurrent
> dd's to the separate drives could show whether it's the RAID that's
> slowing things down.
Ok, I did run the following dd command in different combinations:

   dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000

Here the results:

   Each disk alone
   /dev/sdc1 59.094636 MB/s
   /dev/sdd1 58.686592 MB/s
   /dev/sde1 55.282807 MB/s
   /dev/sdf1 62.271240 MB/s
   /dev/sdg1 60.872891 MB/s
   /dev/sdh1 62.252781 MB/s
   /dev/sdi1 59.145637 MB/s
   /dev/sdj1 60.921119 MB/s

   sdc + sdd in parallel (2 disks on same channel)
   /dev/sdc1 42.512287 MB/s
   /dev/sdd1 43.118483 MB/s

   sdc + sdg in parallel (2 disks on different channels)
   /dev/sdc1 42.938186 MB/s
   /dev/sdg1 43.934779 MB/s

   sdc + sdd + sde in parallel (3 disks on same channel)
   /dev/sdc1 35.043501 MB/s
   /dev/sdd1 35.686878 MB/s
   /dev/sde1 34.580457 MB/s

   Similar results for three disks (sdg + sdh + sdi) on the other channel
   /dev/sdg1 36.381137 MB/s
   /dev/sdh1 37.541758 MB/s
   /dev/sdi1 35.834920 MB/s

   sdc + sdd + sde + sdf in parallel (4 disks on same channel)
   /dev/sdc1 31.432914 MB/s
   /dev/sdd1 32.058752 MB/s
   /dev/sde1 31.393455 MB/s
   /dev/sdf1 33.208165 MB/s

   And here for the four disks on the other channel
   /dev/sdg1 31.873028 MB/s
   /dev/sdh1 33.277193 MB/s
   /dev/sdi1 31.910000 MB/s
   /dev/sdj1 32.626744 MB/s

   All 8 disks in parallel
   /dev/sdc1 24.120545 MB/s
   /dev/sdd1 24.419801 MB/s
   /dev/sde1 24.296588 MB/s
   /dev/sdf1 25.609548 MB/s
   /dev/sdg1 24.572617 MB/s
   /dev/sdh1 25.552590 MB/s
   /dev/sdi1 24.575616 MB/s
   /dev/sdj1 25.124165 MB/s

So from these results, I may assume that md is not the cause of the problem.

What comes as a big surprise is that I loose 25% performance with only
two disks and each hanging on its own channel!

Is this normal? I wonder if other people have the same problem with
other controllers or the same.

What can I do next to find out if this is a kernel, driver or hardware
problem?

Thanks,
Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/