[RFC PATCH net-next v1 0/2] net: per-socket opt-out from RFS/aRFS

[Vitaly Zuevsky]

RFS and accelerated RFS are enabled per host (rps_sock_flow_entries)
and per rx queue (rps_flow_cnt, NETIF_F_NTUPLE). Whether steering a
given flow is worth it is a property of the flow, and a process using
the flows of different traffic patterns cannot express it.

Motivating workload: a CDN edge running nginx with workers and IRQs
pinned, with deliberately more NIC queue -> CPU lanes than workers.
Outbound connections to peers and origins are long-lived and benefit
from aRFS. The larger population of client connections is
short-lived, arrives on an RSS-chosen CPU and is served from there;
steering them gains nothing, yet each is recorded on its first
recvmsg() and pushed into the NIC via ndo_rx_flow_steer(). On mlx5
that is a GFP_ATOMIC allocation and a per-device spinlock in softirq,
a firmware command on a single-threaded workqueue, and an O(rules)
expiry walk under the same lock. Enabling aRFS on such a host more
than doubled softirq CPU and, as traffic grew, ended in softirq
livelock and a forced reboot. Aligning accept() CPUs with RSS through
reuseport does not apply because the queue:worker mapping is
intentionally not 1:1.

This series adds SO_NO_RFS, a boolean socket option. When set,
sock_rps_record_flow() does not record the socket's flow in
rps_sock_flow_table, so get_rps_cpu() finds no match and neither
software RFS nor aRFS acts on the flow; it stays on the CPU chosen by
RSS (plain RPS is per queue and unaffected). The bit lives in sk_flags
and is therefore copied by sk_clone_lock(): set once on a listening
socket, every accept()ed child is excluded, while connect()ed sockets
of the same process keep RFS/aRFS.

Cost on the record path is one sock_flag() test, reached only when
the rfs_needed static key is on and sk_state == TCP_ESTABLISHED;
sk_flags is already read by tcp_recvmsg() so the line is hot.

Status: compile-tested on x86_64 with CONFIG_RPS=y and =n, not yet
runtime-tested. Sent as RFC to get agreement on the interface and
semantics first. The "before" state on the motivating host is a
production outage and cannot be re-run; testing on comparable
workloads by others would be very welcome.

Vitaly Zuevsky (2):
  net: add SO_NO_RFS to exclude a socket from RFS and aRFS
  Documentation: networking: document SO_NO_RFS

 Documentation/networking/scaling.rst  | 14 ++++++++++++++
 arch/alpha/include/uapi/asm/socket.h  |  2 ++
 arch/mips/include/uapi/asm/socket.h   |  2 ++
 arch/parisc/include/uapi/asm/socket.h |  2 ++
 arch/sparc/include/uapi/asm/socket.h  |  2 ++
 include/net/rps.h                     | 13 +++++++++++++
 include/net/sock.h                    |  1 +
 include/uapi/asm-generic/socket.h     |  2 ++
 net/core/sock.c                       |  8 ++++++++
 9 files changed, 46 insertions(+)


base-commit: 734bef548a96e8774d6eef5dfa2026c6edc4e80b
-- 
2.34.1