Re: designing fast, streaming disk i/o with mmap: help wanted

From: Matti Aarnio (matti.aarnio@sonera.fi)
Date: Sun Apr 02 2000 - 19:36:07 EST


On Fri, Mar 31, 2000 at 02:19:45PM -0500, Paul Barton-Davis wrote:
> I am a significant way along completing a high end audio recorder for
> Linux (high end: 26+ channels of 32 bit audio).
>
> I have been holed up (as well as held up) for the last few weeks
> trying to design a new buffering scheme for the data after I realized
> that my old one had a fatal application-level design error.
>
> I thought I had finished the new design, but have found that it
> doesn't perform as well as the old one, and I suspect that this is
> partly to do with the different way I am using mmap(2) in the new one.

        Read-ahead (there is no write ahead) at mmap(2) is constrained
        to ONE instance per entire process memory space.

        That alone makes use of mmap(2) rather inefficient, doesn't it ?

> The application level requirements: suppose that we have a file of
> recorded data. We need to read-ahead a certain amount to ensure that
> the audio thread never blocks waiting for data to come from the disk
> (seek time issues). At the same time, certain tracks, from time to
> time, may have data recorded to them that needs to be written back to
> disk. Note that this is *not* a bi-state model: all tracks must always
> be read-ahead, some tracks have recorded data.

        A separate process thread along with separate FD to same
        file for reading and writing will give you the read-ahead
        you are looking for, and parallel to that, writing.

        using aio_*() might be usefull too.

...
> Overall, we are looking at requiring up to 9MB/sec effective streaming
> rate (visible only after buffering in the ringbuffer) both to and from
> the disk (18MB/sec total throughput) for 24 tracks.
>
> Its not clear to me how best to utilize the interaction between
> mmap(2) and the buffer cache to get optimal performance for this kind
> of thing. I would welcome any comments or ideas.

        Having the process mlock()ed into the core helps by making sure that
        it won't slip away to paging when time-critical thing is underway.

> --p

/Matti Aarnio <matti.aarnio@sonera.fi>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Fri Apr 07 2000 - 21:00:09 EST