Re: [RFC PATCH net-next 0/4] tcp: carry a BPF-chosen identifier in TSval low bits

[Stanislav Fomichev]

On 10/13, Willem Ferguson wrote:
> This series lets a listener encode a small per-connection identifier in
> the low bits of the TCP timestamp values it sends. The peer echoes TSval
> back in TSecr, so every segment the peer sends after the handshake
> carries the identifier in its TSecr.
> 
> In Anycast networks, mid-stream path changes caused by BGP route flapping
> or ECMP shifts frequently deliver active TCP packets to an ingress Point
> of Presence (PoP) that lacks the flow's socket state, resulting in abrupt
> connection resets (RST). While protocols like QUIC avoid this using explicit
> Connection IDs, TCP lacks an in-band routing token to help ingress
> nodes identify where a flow actually lives. To address this problem,
> we propose embedding an N-bit data center ID into the lower-order bits
> of the TCP Timestamp option (TSval, RFC 7323), allowing edge load balancers
> to recognize misdirected packets and steer them to the correct origin data
> center holding the active session.
> 
> How it works:
> 
>  - tcp_ts_id_bits (per netns, 0..8, default 0) sets how many low TSval
>    bits are reserved. The millisecond clock is shifted up by that many
>    bits.
>  - A sock_ops program opts a listener in with
>    BPF_SOCK_OPS_TCP_TS_ID_CB_FLAG and is called with
>    BPF_SOCK_OPS_TCP_TS_ID_CB for each SYN, before the SYN-ACK is sent.
>    It sees the SYN and optionally returns the identifier
>  - The identifier and width are stored in the request socket, then in
>    the full and timewait sockets. Incoming TSecr values are decoded
>    before RTT estimation and PAWS use them.
>  - TCP_TS_ID reads back the identifier, and /proc/net/netstat gains
>    TCPTSIdAssigned, TCPTSIdDeclined, TCPTSIdSkipped and
>    TCPTSIdEchoMismatch.
> 
> RTT estimation on the encoding host sees the same decoded timestamps
> as before, so they are unchanged. On the peer, PAWS keeps working too;
> the only effects are a shorter idle limit and a narrower tolerance for
> reordered segments, both described below.
> 
> Scope and limitations:
> 
>  - Passive opens only. SYN cookies, microsecond timestamps and peers
>    without timestamps are left unencoded.
>  - The peer sees a clock running 2^bits times faster. That narrows the
>    real-time span of its PAWS window, so the width should be no larger
>    than the identifiers need.
>  - Peers that keep ts_recent for the RFC 7323 24 days (e.g. FreeBSD,
>    macOS) stall connections that stay idle longer than 24.8/2^bits days
>    (about 37 hours at 4 bits). Linux peers stop enforcing PAWS after
>    TCP_PAWS_WRAP and are unaffected.
>  - TCP_REPAIR cannot restore the identifier, so hosts that migrate
>    established connections should leave the feature off.
> 
> All of this is documented under tcp_ts_id_bits in ip-sysctl.rst. With
> the sysctl at 0 (the default) the only new cost on the established fast
> path is one likely() branch when parsing TSecr.
> 
> Questions for reviewers:
> 
>  1. Is encoding an identifier in the low bits of TSval acceptable in
>     principle, given the trade-offs above?
>  2. If so, what feedback do you have on the implementation?

Don't see any feedback yet, so from the top of my head: looks like you
control two sides of the communication, so why not a custom (bpf) tcp option?
Would you need that on every packet and it makes it expensive?

Or any other field you can abuse maybe? IPv6 Flow label?

Implementation-wise, the sock_ops is going away in favor of bpf_tcp_ops,
so BPF_SOCK_OPS_TCP_TS_ID_CB_FLAG is a no go.