Re: [fuse-devel] fuse: max_background and congestion_threshold settings

From: Maxim Patlasov
Date: Wed Nov 16 2016 - 17:14:22 EST

On 11/16/2016 12:19 PM, Nikolaus Rath wrote:

On Nov 16 2016, Maxim Patlasov <mpatlasov@xxxxxxxxxxxxx> wrote:
On 11/16/2016 11:19 AM, Nikolaus Rath wrote:

Hi Maxim,

On Nov 15 2016, Maxim Patlasov <mpatlasov@xxxxxxxxxxxxx> wrote:
On 11/15/2016 08:18 AM, Nikolaus Rath wrote:
Could someone explain to me the meaning of the max_background and
congestion_threshold settings of the fuse module?

At first I assumed that max_background specifies the maximum number of
pending requests (i.e., requests that have been send to userspace but
for which no reply was received yet). But looking at fs/fuse/dev.c, it
looks as if not every request is included in this number.
fuse uses max_background for cases where the total number of
simultaneous requests of given type is not limited by some other
natural means. AFAIU, these cases are: 1) async processing of direct
IO; 2) read-ahead. As an example of "natural" limitation: when
userspace process blocks on a sync direct IO read/write, the number of
requests fuse consumed is limited by the number of such processes
(actually their threads). In contrast, if userspace requests 1GB
direct IO read/write, it would be unreasonable to issue 1GB/128K==8192
fuse requests simultaneously. That's where max_background steps in.
Ah, that makes sense. Are these two cases meant as examples, or is that
an exhaustive list? Because I would have thought that other cases should
be writing of cached data (when writeback caching is enabled), and
asynchronous I/O from userspace...?
I think that's exhaustive list, but I can miss something.

As for writing of cached data, that definitely doesn't go through
background requests. Here we rely on flusher: fuse will allocate as
many requests as the flusher wants to writeback.

Buffered AIO READs actually block in submit_io until fully
processed. So it's just another example of "natural" limitation I told
Not sure I understand. What is it that's blocking? It can't be the
userspace process, because then it wouldn't be asynchronous I/O...

Surprise! Alas, Linux kernel does NOT process buffered AIO reads in async manner. You can verify it yourself by strace-ing a simple program looping over io_submit + io_getevents: for direct IO (as expected) io_submit returns immediately while io_getevents waits for actual IO; in contrast, for buffered IO (surprisingly) io_submit waits for actual IO while io_getevents returns immediately. Presumably, people are supposed to use mmap-ed read/writes rather than buffered AIO.

Also, I am not sure what you mean with async processing of direct
I/O. Shouldn't direct I/O always go directly to the file-system? If so,
how can it be processed asynchronously?
That's a nice optimization we implemented a few years ago: having
incoming sync direct IO request of 1MB size, kernel fuse splits it
into eight 128K requests and starts processing them in async manner,
waiting for the completion of all of them before completing that
incoming 1MB requests.
I see. But why isn't that also done for regular (non-direct) IO?

Regular READs are helped by async read-ahead. Regular writes go through writeback mechanics: flusher calls fuse_writepages() and the latter submits as many async write requests as needed. Everything looks fine. (but as I wrote those async requests are not under fuse max_backgroung control).