Re: [PATCH 0/2] dm-ioband: I/O bandwidth controller v1.7.0: Introduction
From: Dong-Jae Kang
Date: Thu Oct 09 2008 - 02:15:31 EST
Hi, Ryo tsuruta.
Thank you for your fast reply.
Your comments was very helpful for me ^^
> 2. tiotest is not an appropriate tool to see how bandwith is shared
> among devices, becasue those three tiotests don't finish at the
> same time, a process which issues I/Os to a device with the highest
> weight finishes first, so you can't see how bandwidth is shared
> from the results of each tiotest.
Yes, you are right, and it is good point for correct IO testing in
dm-ioband and other controllers.
So, I tested dm-ioband and bio-cgroup patches with another IO testing
tool, xdd ver6.5(http://www.ioperformance.com/), after your reply.
Xdd supports O_DIRECT mode and time limit options.
I think, personally, it is proper tool for testing of IO controllers
in Linux Container ML.
And I found some strange points in test results. In fact, it will be
not strange for other persons^^
1. dm-ioband can control IO bandwidth well in O_DIRECT mode(read and
write), I think the result is very reasonable. but it can't control it
in Buffered mode when I checked just only output of xdd. I think
bio-cgroup patches is for solving the problems, is it right? If so,
how can I check or confirm the role of bio-cgroup patches?
2. As showed in test results, the IO performance in Buffered IO mode
is very low compared with it in O_DIRECT mode. In my opinion, the
reverse case is more natural in real life.
Can you give me a answer about it?
3. Compared with physical bandwidth(when it is checked with one
process and without dm-ioband device), the sum of the bandwidth by
dm-ioband has very considerable gap with the physical bandwidth. I
wonder the reason…. Is it overhead of dm-ioband or bio-cgroup patches?
or Are there any another reasons?
the new testing result is like below.
- Testing target : the patches of dm-ioband ver1.7.0 and bio-cgroup
latest version
- Testing Cases
1.Read and write stress test in O_DIRECT IO mode
2.Read and write stress test in not Buffered IO mode
- Testing tool : xdd ver6.5 ( http://www.ioperformance.com/ )
* Total bandwidth
Read IO
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 1048576000 128000 15.700 66.790 8153.04 0.0001
0.00 read 8192
0 1 1048576000 128000 15.700 66.790 8153.04 0.0001
0.00 read 8192
1 1 1048576000 128000 15.700 66.790 8153.04 0.0001
0.00 read 8192
Write IO
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 1048576000 128000 14.730 71.185 8689.59 0.0001
0.00 write 8192
0 1 1048576000 128000 14.730 71.185 8689.59 0.0001
0.00 write 8192
1 1 1048576000 128000 14.730 71.185 8689.59 0.0001
0.00 write 8192
* Read IO test in O_DIRECT mode
Command :
xdd.linux -op read -targets 1 /dev/mapper/ioband1 -reqsize 8 -numreqs
128000 -verbose -timelimit 30 –dio
Result :
cgroup1 (weight : 10)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 84549632 10321 30.086 2.810 343.05 0.0029
0.00 read 8192
0 1 84549632 10321 30.086 2.810 343.05 0.0029
0.00 read 8192
1 1 84549632 10321 30.086 2.810 343.05 0.0029
0.00 read 8192
cgroup1 (weight : 30)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 256425984 31302 30.089 8.522 1040.31 0.0010
0.00 read 8192
0 1 256425984 31302 30.089 8.522 1040.31 0.0010
0.00 read 8192
1 1 256425984 31302 30.089 8.522 1040.31 0.0010
0.00 read 8192
cgroup1 (weight : 60)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 483467264 59017 30.000 16.116 1967.22 0.0005
0.00 read 8192
0 1 483467264 59017 30.000 16.116 1967.22 0.0005
0.00 read 8192
1 1 483467264 59017 30.000 16.116 1967.22 0.0005
0.00 read 8192
* Write IO test in O_DIRECT mode
Command :
xdd.linux -op write -targets 1 /dev/mapper/ioband1 -reqsize 8 -numreqs
128000 -verbose -timelimit 30 –dio
Result :
cgroup1 (weight : 10)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 106790912 13036 30.034 3.556 434.04 0.0023
0.00 write 8192
0 1 106790912 13036 30.034 3.556 434.04 0.0023
0.00 write 8192
1 1 106790912 13036 30.034 3.556 434.04 0.0023
0.00 write 8192
cgroup1 (weight : 30)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 347176960 42380 30.006 11.570 1412.40 0.0007
0.00 write 8192
0 1 347176960 42380 30.006 11.570 1412.40 0.0007
0.00 write 8192
1 1 347176960 42380 30.006 11.570 1412.40 0.0007
0.00 write 8192
cgroup1 (weight : 60)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 636256256 77668 30.000 21.209 2588.93 0.0004
0.00 write 8192
0 1 636256256 77668 30.000 21.209 2588.93 0.0004
0.00 write 8192
1 1 636256256 77668 30.000 21.209 2588.93 0.0004
0.00 write 8192
* Read IO test in Buffered IO mode
Command :
xdd.linux -op read -targets 1 /dev/mapper/ioband1 -reqsize 8 -numreqs
128000 -verbose -timelimit 30
Result :
cgroup1 (weight : 10)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 161284096 19688 30.012 5.374 656.00 0.0015
0.00 read 8192
0 1 161284096 19688 30.012 5.374 656.00 0.0015
0.00 read 8192
1 1 161284096 19688 30.012 5.374 656.00 0.0015
0.00 read 8192
cgroup1 (weight : 30)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 162816000 19875 30.005 5.426 662.38 0.0015
0.00 read 8192
0 1 162816000 19875 30.005 5.426 662.38 0.0015
0.00 read 8192
1 1 162816000 19875 30.005 5.426 662.38 0.0015
0.00 read 8192
cgroup1 (weight : 60)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 167198720 20410 30.002 5.573 680.29 0.0015
0.00 read 8192
0 1 167198720 20410 30.002 5.573 680.29 0.0015
0.00 read 8192
1 1 167198720 20410 30.002 5.573 680.29 0.0015
0.00 read 8192
* Write IO test in Buffered IO mode
Command :
xdd.linux -op write -targets 1 /dev/mapper/ioband1 -reqsize 8 -numreqs
128000 -verbose -timelimit 30
Result :
cgroup1 (weight : 10)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 550633472 67216 30.017 18.344 2239.30 0.0004
0.00 write 8192
0 1 550633472 67216 30.017 18.344 2239.30 0.0004
0.00 write 8192
1 1 550633472 67216 30.017 18.344 2239.30 0.0004
0.00 write 8192
cgroup1 (weight : 30)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 32768 4 32.278 0.001 0.12 8.0694
0.00 write 8192
0 1 32768 4 32.278 0.001 0.12 8.0694
0.00 write 8192
1 1 32768 4 32.278 0.001 0.12 8.0694
0.00 write 8192
cgroup1 (weight : 60)
T Q Bytes Ops Time Rate IOPS Latency
%CPU OP_Type ReqSize
0 1 4505600 550 31.875 0.141 17.25 0.0580
0.00 write 8192
0 1 4505600 550 31.875 0.141 17.25 0.0580
0.00 write 8192
1 1 4505600 550 31.875 0.141 17.25 0.0580
0.00 write 8192
> I use iostat to see the time variation of bandiwdth. The followings
> are the outputs of iostat just after starting three tiotests on the
> same setting as yours.
>
> # iostat -p dm-0 -p dm-1 -p dm-2 1
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> dm-0 5430.00 0.00 10860.00 0 10860
> dm-1 16516.00 0.00 16516.00 0 16516
> dm-2 32246.00 0.00 32246.00 0 32246
>
> avg-cpu: %user %nice %system %iowait %steal %idle
> 0.51 0.00 21.83 76.14 0.00 1.52
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> dm-0 5720.00 0.00 11440.00 0 11440
> dm-1 16138.00 0.00 16138.00 0 16138
> dm-2 32734.00 0.00 32734.00 0 32734
> ...
>
Thank you for your kindness ^^
>
> Could you give me the O_DIRECT patch?
>
Of course, if you want. But it is nothing
Tiobench tool is very simple and light source code, so I just add the
O_DIRECT option in tiotest.c of tiobench testing tool.
Anyway, after I make a patch file, I send it to you
Best Regards,
Dong-Jae Kang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/