Re: tcp_sendpage and page allocation lifetime vs. iscsi

From: Avi Kivity
Date: Mon Apr 25 2005 - 14:25:55 EST


Olivier Galibert wrote:

I have a problem with the iscsi driver (both 4.x and 5.x) and scsi
tape I'm not sure how to solve. It may linked to some specific
characteristics of the tg3 network driver.

What happens is, from what I can trace:
1- st alloc_pages a bunch of pages for buffering

2- st sends a bunch of them to iscsi for writing (32K is common when
labelling a tape for instance)

3- iscsi sends whatever header is needed followed by the data using
tcp_sendpage

4- tcp_sendpage copies from of the pages but get_page() others,
probably depending on the state of the socket buffer. It returns
immediatly anyway, leaving some pages with an elevated count (which, I
guess, it will eventually decrement again)

5- iscsi returns to st

6- st reuses the buffer immediatly, and/or frees it if the device is
closed. Silent corruption in one case, bad_page in __free_page_ok
called from normalize_buffer in the other.

I'm going to complete my traces to be sure that's really what's going
on (I don't have a log immediatly after sendpage yet). But in any
case, what would the solution be?



you need a completion to tell you when your buffer has been sent. you can use the kiocb parameter to tcp_sendmsg, as it has a completion. however, tcp_sendmsg does not appear to use it.

in effect, you need tcp aio, but the mainline kernel does not support it yet.

--
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/