Re: [RFC] splice: add wait-all semantics for socket-to-pipe reads

[Kuniyuki Iwashima]

On Tue, Sep 1, 2026 at 8:53 PM 天狼 <rockeet@xxxxxxxxx> wrote:
>
> Hi,
>
> I would like to ask whether splice(2) could gain wait-all semantics for
> blocking socket-to-pipe transfers, analogous to recv(MSG_WAITALL).
>
> The concrete use case is a high-performance S3 FUSE client.  Once an HTTP
> response header has been parsed, the response body length and the exact range
> requested by FUSE are known.  The body is moved from a TCP socket into a pipe
> used for an fd-backed FUSE reply.  No userspace data buffer is needed.
>
> Today userspace has to loop around splice() until the requested length has
> been transferred.  On a fast local network this commonly turns one logical
> read into several splice syscalls and scheduling transitions.  In one recent
> profile, 3,475 S3 GET requests caused 25,463 splice syscalls (7.33 per GET).
> The profile contained no poll or setsockopt calls in the hot path, and 80.86%
> of the sampled CPU time was in the kernel.
>
> The obvious workarounds all have undesirable properties:
>
>   - looping over splice() in userspace adds syscalls and wakeups for short
>     positive returns;
>   - poll plus SO_RCVLOWAT waits for data to accumulate in the socket receive
>     buffer, adds another syscall, and couples latency to an advisory readiness
>     threshold;
>   - recv(MSG_WAITALL) requires a userspace buffer and loses the socket-to-pipe
>     splice path;
>   - increasing SO_RCVBUF does not make splice wait for the requested length.
>
> The current TCP implementation is already close to providing the desired
> behavior.  tcp_splice_read() obtains its time budget with sock_rcvtimeo() and
> loops while data remains immediately available.  However, after making any
> positive progress, a later iteration that finds no data returns immediately
> because of:
>
>         else if (!ret) {
>                 if (spliced)
>                         break;
>                 ...
>                 ret = sk_wait_data(sk, &timeo, NULL);
>         }
>
> Would a new flag such as SPLICE_F_WAITALL be acceptable?

The use case makes sense, but I think it depends on the numbers.

Another aspect would be whether we want to add a new (potentially
buggy) feature for a single user in this AI era.

It would be great if the change is proven to improve performance
of the official client that has a lot more users worldwide.
https://github.com/awslabs/mountpoint-s3/


>  For a blocking
> socket-to-pipe splice it would mean:
>
>   - keep transferring until len bytes have entered the pipe;
>   - after a short positive transfer followed by temporary lack of input,
>     wait in the kernel with sk_wait_data() instead of returning;
>   - use the socket's existing SO_RCVTIMEO as the total receive wait budget;
>   - still return early for EOF, shutdown, a pending socket error, or a signal;
>   - on timeout/error/signal after partial progress, return the partial byte
>     count, matching the usual stream-receive convention; otherwise return the
>     corresponding error;
>   - retain the existing behavior when SPLICE_F_WAITALL is absent.
>
> SPLICE_F_NONBLOCK/O_NONBLOCK would remain orthogonal: WAITALL would not cause
> sleeping on a nonblocking descriptor.  The destination pipe must either have
> enough capacity for len or be drained concurrently, as with blocking splice
> today.  For the motivating workload the pipe is sized for the FUSE request.
>
> The important property is that data can be spliced into the pipe immediately
> as TCP data arrives.  Unlike poll plus SO_RCVLOWAT, this does not require the
> whole range to accumulate in the socket buffer before transfer begins, while
> still keeping the wait-and-retry loop inside one syscall.
>
> This appears implementable as a small conditional change in the TCP splice
> read loop plus UAPI documentation and selftests.  I am sending this RFC first
> to confirm whether a splice flag is the right interface and whether the
> SO_RCVTIMEO-based partial-return semantics are acceptable before preparing a
> patch.
>
> The userspace implementation and CI profile that motivated this request are
> available here:
>
>   https://github.com/topling/ngs3fs/commit/4b8f5a69ceebd9211021a5d11496373d63f2e43d
>   https://github.com/topling/ngs3fs/actions/runs/33586457571
>
> Thanks,
> leipeng