Re: fio posixaio performance problem

[Shaohua Li]

在 2011年8月4日 上午10:00，Gui Jianfeng <guijianfeng@xxxxxxxxxxxxxx> 写道：
> On 2011-8-4 8:53, Shaohua Li wrote:
>> 2011/8/4 Vivek Goyal <vgoyal@xxxxxxxxxx>:
>>> On Wed, Aug 03, 2011 at 11:45:33AM -0400, Vivek Goyal wrote:
>>>> On Wed, Aug 03, 2011 at 05:48:54PM +0800, Gui Jianfeng wrote:
>>>>> On 2011-8-3 16:22, Shaohua Li wrote:
>>>>>> 2011/8/3 Gui Jianfeng <guijianfeng@xxxxxxxxxxxxxx>:
>>>>>>> On 2011-8-3 15:38, Shaohua Li wrote:
>>>>>>>> 2011/8/3 Gui Jianfeng <guijianfeng@xxxxxxxxxxxxxx>:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I ran a fio test to simulate qemu-kvm io behaviour.
>>>>>>>>> When job number is greater than 2, IO performance is
>>>>>>>>> really bad.
>>>>>>>>>
>>>>>>>>> 1 thread: aggrb=15,129KB/s
>>>>>>>>> 4 thread: aggrb=1,049KB/s
>>>>>>>>>
>>>>>>>>> Kernel: lastest upstream
>>>>>>>>>
>>>>>>>>> Any idea?
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> [global]
>>>>>>>>> runtime=30
>>>>>>>>> time_based=1
>>>>>>>>> size=1G
>>>>>>>>> group_reporting=1
>>>>>>>>> ioengine=posixaio
>>>>>>>>> exec_prerun='echo 3 > /proc/sys/vm/drop_caches'
>>>>>>>>> thread=1
>>>>>>>>>
>>>>>>>>> [kvmio-1]
>>>>>>>>> description=kvmio-1
>>>>>>>>> numjobs=4
>>>>>>>>> rw=write
>>>>>>>>> bs=4k
>>>>>>>>> direct=1
>>>>>>>>> filename=/mnt/sda4/1G.img
>>>>>>>> Hmm, the test runs always about 15M/s at my side regardless how many threads.
>>>>>>>
>>>>>>> CFQ？
>>>>>> yes.
>>>>>>
>>>>>>> what's the slice_idle value?
>>>>>> default value. I didn't change it.
>>>>>
>>>>> Hmm, I use a sata disk, and can reproduce this bug every time...
>>>>
>>>> Do you have blktrace of run with 4 jobs?
>>>
>>> I can't reproduce it too. On my sata disk single thread is getting around
>>> 23-24MB/s and 4 threads get around 19-20MB/sec. Some of the throughput
>>> is gone into seeking so that is expected.
>>>
>>> I think what you are trying to point out is idling issue. In your workload
>>> every thread is doing sync-idle IO. So idling is enabled on each thread.
>>> On my system I see that next thread preempts the current idle thread
>>> because they all are doing IO in nearby area of file and rq_close() is
>>> true hence preemption is allowed.
>>>
>>> On your system, I think somehow rq_close() is not true hence preemption
>>> does not take place and we continue to idle on that thread. That also
>>> is not necessarily too bad but it might be happening that we are waiting
>>> for completion of IO from some other thread before this thread (we are
>>> idling on) can do more writes due to some filesystem rescrition and
>>> that can lead to sudden throughput drop. blktrace will give some idea.
>> with idle, the workload fallbacks like the one thread case, I don't
>> expect so big reduction.
>> I saw some back seek in the workload because we have rq_close() preempt here.
>> is it possible back seek penality in the disk is big?
>
> Shaohua,
>
> what do you mean "back seek penality" here. AFAIK, back seek penality only happens
> when choosing next request to serve. Is there anything to do with preemption logic?
oh, not related per your blktrace. so we have two problems here:
1. fio doesn't dispatch request in 8ms.
2. no close request preempt.
both looks quite wield. can you post a longer blktrace output, like
for one second? the piece is too short.
and do you have anything others running?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/