Re: fio rand read/write regression with 2.6.32-rc3

[Zhang, Yanmin]

On Sun, 2009-10-11 at 10:23 +0200, Corrado Zoccolo wrote:
> On Sat, Oct 10, 2009 at 11:52 AM, Zhang, Yanmin
> <yanmin_zhang@xxxxxxxxxxxxxxx> wrote:
> > On Sat, 2009-10-10 at 10:01 +0200, Jens Axboe wrote:
> >> On Sat, Oct 10 2009, Zhang, Yanmin wrote:
> >> > Comparing with 2.6.23-rc1's result, fio rand read write has regression
> >> > on my 2*4 core stoakley machine (8GB memory) with a JBOD of 12 disks.
> >> >
> >> > Every disk has 8 1-GB files. Start 8 sub-processes per disk and every
> >> > process random chooses a file on the disk to do 36 times of file read or
> >> > write on the file and then choose another file.
> >> >
> >> >
> >> > fio_mmap_rand_read_4k regresion is about 35%.
> >>
> >> Heh, I seem to recollect I told Linus that this would cost is 30-40%
> >> performance. So not totally crazy.
> >>
> >> So yes, this isn't hugely unexpected. If you send me your fio job files,
> >> I'll try and see what I can do about it.
> > See the attachment.
> >
> Hi Yanmin,
> the fio test you sent just performs random read, no write seems involved here.
ï
Sorry for not sending all the job files because they are similiar and big.
You could change the job file to run rand write and readwrite.

> I suspect that you should be able to observe the same regression if
> you just run on a single disk. Can you confirm?
> Is your disk a SATA2 rotational disk with NCQ?
It's a JOBD with 12 SAS disks.

Another machine's JBOD has 13 SCSI disks.

Based on your suggestion, I tested it on a single SAS disk (create 8 1-GB file and
start 8 processes) and get 30% regression. On a single SCSI disk, there is also about
30% regression. On a single SATA (SATA link up 1.5 Gbps) disk with NCQ, the regression
is about 20%.

Perhaps the job file causes fio to deliver too many I/O requests, sometimes kernel
print out some blocking info like below:

Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: INFO: task kjournald:6261 blocked for more than 120 seconds.
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: kjournald     D ffff88002f228180     0  6261      2
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  ffff88027ab09910 0000000000000046 0000000000000000 ffff88027cc20440
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  ffff88027f0c66a0 ffff88027ab09bb0 0000000100000e56 000000007cc20440
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  ffff88027a915480 ffffffff803678a9 0000000000000000 0000000100000000
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel: Call Trace:
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  [<ffffffff803678a9>] ? __make_request+0x310/0x40b
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  [<ffffffff802b5571>] ? sync_buffer+0x0/0x3f
Apr  8 23:54:19 lkp-tulsa01-x8664 kernel:  [<ffffffff805aeadf>] ? schedule+0x9/0x1d


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/