Re: Networking Question

[Ryan P. Nicholl]

Thanks for taking time to respond to this question, but unfortunately Linux's shutdown(2) cannot do what I need. This isn't a general question since POSIX has no standard async API other than AIO which isn't implemented efficiently in GNU/Linux, and I don't think the kernel supports any aio calls directly, instead exposing clone and epoll_* facilities to solve the concurrency issues. However; none of them do what I need efficiently.

In theory, shutdown(2) could be used, the problem is that it's just not efficiently scalable to 100s of thousands or millions of connections. I'm not speaking on behalf of my employeer, but I work for a financial company that processes a lot of network traffic.

What I want essentially is similar to TIME_WAIT but instead of sending an RST packet I want to be notified when the tcp connection is actually drained.

If you call shutdown, my understanding is that you get one of two behaviors:
1. You send RST and immediately discard the send buffer.
2. The call blocks for the SO_LINGER timeout.

Option 1 isn't acceptable because it gives the wrong behavior. Option 2 can be made to work but the downside is that Linux only exposes a synchronous API for it which requires me to have at least 1 thread per shutdown operation. So it could be quite bad if the network is very congested. Unfortunately threads are many times more expensive than sockets/tcp connections. So ideally having few threads and many sockets gives the best use of resources.



--
Ryan P. Nicholl


------- Original Message -------
On Tuesday, June 21st, 2022 at 3:02 AM, Bernd Petrovitsch <bernd@xxxxxxxxxxxxxxxxxxx> wrote:


> Hi all!
>
> On 21/06/2022 02:29, Ryan P. Nicholl wrote:
>
> > I've been unable to find any Linux API for asynchronously waiting for the TCP send buffer to drain.
> >
> > The problem I have, in a nutshell, is noted in this part of the documentation:
> >
> > If fildes refers to a socket, close() shall cause the socket to
> > be destroyed. If the socket is in connection-mode, and the
>
>
> That's not really a Linux kernel question as such (because that should
> work that way on all TCP connections anywhere) but the shutdown()
> syscall is probably what you need:
> - your side shuts down the sending part of the socket.
> - the other side reads data and gets eventually EOF
> - the other side call shutdown() for it's sending side when it's done.
> - your side gets EOF.
> And then your side knows that no data is in flight.
> - finally, you clean up with close(). You can shutdown() the receiving
> side too but doesn't change anything.
>
> [ deleted SO_LINGER stuff - that's for something completelly different ... ]
>
> Kind regards,
> Bernd
> --
> Bernd Petrovitsch Email : bernd@xxxxxxxxxxxxxxxxxxx
> There is NO CLOUD, just other people's computers. - FSFE
> LUGA : http://www.luga.at