async I/O seems to be blocking on 2.6.15

From: Brent Baccala
Date: Fri Nov 03 2006 - 03:23:57 EST


Hello -

I'm running 2.6.15 (Debian) on a Pentium M laptop, PCI attached ext3
filesystem.

I'm writing my first asynchronous I/O program, and for a while I
thought I was really doing something wrong, but more and more I'm
starting to conclude that the problem might be in the kernel.

Basically, I've narrowed things down to a test program which opens a
large (700 MB) file in O_DIRECT mode and fires off 100 one MB async
reads for the first 100 MB of data. The enqueues take about 5 seconds
to complete, which is also about the amount of time this disk needs to
read 100 MB, so I suspect that it's blocking.

I've gotten the POSIX AIO interface at least tolerably running using
the GLIBC thread-based implementation, but I really want the native
interface working.

I whittled the test program down to use system calls instead of the
POSIX AIO library, and I'm attaching a copy. You put a big file at
'testfile' (it just reads it) and run the program:


baccala@debian ~/src/endgame$ time ./testaio
Enqueues starting
Enqueues complete

real 0m5.327s
user 0m0.004s
sys 0m0.740s
baccala@debian ~/src/endgame$


Of that five seconds, it's almost all spent between the two "enqueues"
messages.

If anybody can shed any light on this, I'd appreciate your feedback
direct to cosine@xxxxxxxxxxxx (I don't read the list).

Thank you.



-bwb

Brent Baccala
cosine@xxxxxxxxxxxx
#define _GNU_SOURCE /* to get O_DIRECT */

#include <stdio.h>
#include <stdlib.h>
#include <string.h> /* for memset() */
#include <unistd.h> /* for _PC_REC_XFER_ALIGN */
#include <asm/unistd.h>
#include <fcntl.h>
#include <linux/aio_abi.h>
#include <errno.h>

_syscall2(int, io_setup, int, maxevents, aio_context_t *, ctxp)
_syscall3(int, io_submit, aio_context_t, ctx, long, nr, struct iocb **, iocbs)

#define NUMAIOS 100

#define BUFFER_BYTES (1<<20)

struct iocb iocb[NUMAIOS];
void *buffer[NUMAIOS];

aio_context_t aio_default_context;

main()
{
int fd;
int i;
int alignment;
struct iocb * iocbp[1];

fd = open("testfile", O_RDONLY | O_DIRECT);
alignment = fpathconf(fd, _PC_REC_XFER_ALIGN);

for (i=0; i<NUMAIOS; i++) {
if (posix_memalign(&buffer[i], alignment, BUFFER_BYTES) != 0) {
fprintf(stderr, "Can't posix_memalign\n");
}
}

io_setup(1024, &aio_default_context);

fprintf(stderr, "Enqueues starting\n");

for (i=0; i<NUMAIOS; i++) {

memset(&iocb[i], 0, sizeof(struct iocb));

iocb[i].aio_lio_opcode = IOCB_CMD_PREAD;
iocb[i].aio_fildes = fd;
iocb[i].aio_buf = (unsigned long) buffer[i];
iocb[i].aio_nbytes = BUFFER_BYTES;
iocb[i].aio_offset = BUFFER_BYTES * i;
/* aiocb[i].aio_offset = 0; */

iocbp[0] = &iocb[i];
if (io_submit(aio_default_context, 1, iocbp) != 1) {
perror("");
fprintf(stderr, "Can't enqueue aio_read %d\n", i);
}
}

fprintf(stderr, "Enqueues complete\n");
}