async I/O seems to be blocking on 2.6.15

[Brent Baccala]

Hello -

I'm running 2.6.15 (Debian) on a Pentium M laptop, PCI attached ext3
filesystem.

I'm writing my first asynchronous I/O program, and for a while I
thought I was really doing something wrong, but more and more I'm
starting to conclude that the problem might be in the kernel.

Basically, I've narrowed things down to a test program which opens a
large (700 MB) file in O_DIRECT mode and fires off 100 one MB async
reads for the first 100 MB of data.  The enqueues take about 5 seconds
to complete, which is also about the amount of time this disk needs to
read 100 MB, so I suspect that it's blocking.

I've gotten the POSIX AIO interface at least tolerably running using
the GLIBC thread-based implementation, but I really want the native
interface working.

I whittled the test program down to use system calls instead of the
POSIX AIO library, and I'm attaching a copy.  You put a big file at
'testfile' (it just reads it) and run the program:


baccala@debian ~/src/endgame$ time ./testaio
Enqueues starting
Enqueues complete

real    0m5.327s
user    0m0.004s
sys     0m0.740s
baccala@debian ~/src/endgame$


Of that five seconds, it's almost all spent between the two "enqueues"
messages.

If anybody can shed any light on this, I'd appreciate your feedback
direct to cosine@xxxxxxxxxxxx (I don't read the list).

Thank you.



					-bwb

					Brent Baccala
					cosine@xxxxxxxxxxxx
#define _GNU_SOURCE		/* to get O_DIRECT */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>	/* for memset() */
#include <unistd.h>	/* for _PC_REC_XFER_ALIGN */
#include <asm/unistd.h>
#include <fcntl.h>
#include <linux/aio_abi.h>
#include <errno.h>

_syscall2(int, io_setup, int, maxevents, aio_context_t *, ctxp)
_syscall3(int, io_submit, aio_context_t, ctx, long, nr, struct iocb **, iocbs)

#define NUMAIOS 100

#define BUFFER_BYTES (1<<20)

struct iocb iocb[NUMAIOS];
void *buffer[NUMAIOS];

aio_context_t aio_default_context;

main()
{
    int fd;
    int i;
    int alignment;
    struct iocb * iocbp[1];

    fd = open("testfile", O_RDONLY | O_DIRECT);
    alignment = fpathconf(fd, _PC_REC_XFER_ALIGN);

    for (i=0; i<NUMAIOS; i++) {
	if (posix_memalign(&buffer[i], alignment, BUFFER_BYTES) != 0) {
	    fprintf(stderr, "Can't posix_memalign\n");
	}
    }

    io_setup(1024, &aio_default_context);

    fprintf(stderr, "Enqueues starting\n");

    for (i=0; i<NUMAIOS; i++) {

	memset(&iocb[i], 0, sizeof(struct iocb));

	iocb[i].aio_lio_opcode = IOCB_CMD_PREAD;
	iocb[i].aio_fildes = fd;
	iocb[i].aio_buf = (unsigned long) buffer[i];
	iocb[i].aio_nbytes = BUFFER_BYTES;
	iocb[i].aio_offset = BUFFER_BYTES * i;
	/* aiocb[i].aio_offset = 0; */

	iocbp[0] = &iocb[i];
	if (io_submit(aio_default_context, 1, iocbp) != 1) {
	    perror("");
	    fprintf(stderr, "Can't enqueue aio_read %d\n", i);
	}
    }

    fprintf(stderr, "Enqueues complete\n");
}