RE: async I/O seems to be blocking on 2.6.15

From: Chen, Kenneth W
Date: Mon Nov 06 2006 - 16:22:40 EST


Jens Axboe wrote on Sunday, November 05, 2006 4:15 AM
> On Fri, Nov 03 2006, Brent Baccala wrote:
> > On Fri, 3 Nov 2006, Jens Axboe wrote:
> >
> > >Try to time it (visual output of the app is not very telling, and it's
> > >buffered) and then apply some profiling.
> >
> > OK, a little more info. I added gettimeofday() calls after each call
> > to io_submit(), put the timevals in an array, and after everything was
> > done computed the difference between each timeval and the program start
> > time, as well as the deltas. I got this:
> >
> > 0: 0.080s
> > 1: 0.086s 0.006s
> > 2: 0.102s 0.016s
> > 3: 0.111s 0.008s
> > 4: 0.118s 0.007s
> > 5: 0.134s 0.015s
> > 6: 0.141s 0.006s
> > 7: 0.148s 0.006s
> > 8: 0.158s 0.009s
> > 9: 0.164s 0.006s
> > ...
> > 96: 1.036s 0.007s
> > 97: 1.044s 0.007s
> > 98: 1.147s 0.102s
> > 99: 1.155s 0.008s
> >
> > 98 appears to be an aberration. Perhaps three of the times on an
> > average run are around a tenth of a second; all of the others are
> > pretty steady at 7 or 8 microseconds. So, it's basically linear in
> > its time consumption.
> >
> > Does 7 microseconds seem a bit excessive for an io_submit (and a
> > gettimeofday)?
>
> I guess you mean miliseconds, not microseconds. 7 miliseconds seems way
> too long. I repeated your test here, and the 100 submits take 97000
> microseconds here - or 97 miliseconds. So that's a little less than 1
> msec per io_submit. Still pretty big. You can experiment with oprofile
> to profile where the kernel spends its time in that period.


I've tried that myself too and see similar result. One thing to note is
that I/O being submitted are pretty big at 1MB, so the vector list inside
bio is going to be pretty long and it will take a while to construct that.
Drop the size for each I/O to something like 4KB will significantly reduce
the time. I haven't done the measurement whether the time to submit I/O
grows linearly with respect to I/O size. Most likely it will. If it is
not, then we might have a scaling problem (though I don't believe we have
this problem).

- Ken


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/