[PATCH net-next v1 02/11] net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP}

[Dave Seddon]

A small targeted optimisation to __skb_flow_dissect(). A new
flow_dissect_fast() dispatcher routes the standard flow_keys dissector
calls through per-shape straight-line extractors for the common
eth + IPv4/IPv6 + TCP/UDP packet shapes. On any miss, falls back to the
existing graph walk. Output is byte-identical to the slow path for the
shapes handled.

This is the parent of the fast-path series. Subsequent patches add
per-shape extensions for single VLAN, QinQ, PPPoE, MPLS, IP-in-IP and
GRE. Each patch ships its own per-shape sysctl in the new
``/proc/sys/net/flow_dissector/`` subtree; this patch establishes the
subtree and registers the first entry, ``eth_ip``.

The fast path is invoked strictly after the netns BPF flow dissector
hook: if a BPF program is attached and returns a verdict other than
BPF_FLOW_DISSECTOR_CONTINUE, __skb_flow_dissect() returns before
flow_dissect_fast() is reached. Attached BPF dissectors therefore
always take precedence and observe no behaviour change from this
series.

  - What this does. Adds an opt-in fast-path for eth + IPv4 (IHL=5,
    not fragmented) + TCP/UDP and eth + IPv6 (no extension headers) +
    TCP/UDP. When the per-shape sysctl is enabled and a packet
    matches, the helper fills struct flow_keys byte-identically to the
    slow path and returns. On any miss it defers to the existing graph
    walk. Only the two standard dissectors are eligible
    (flow_keys_dissector and flow_keys_dissector_symmetric); custom
    dissectors and slow-path-only flags defer unconditionally.

  - How it gates. A new
    DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key), externed in
    <net/flow_dissector.h>, toggled by
    /proc/sys/net/flow_dissector/eth_ip with proc_do_static_key. When
    the sysctl is 0 the dispatcher's per-shape case is a forward not-
    taken JMP — same slow-path cost as today. When 1, the helper runs
    inline.

  - Namespace. The sysctl subtree is /proc/sys/net/flow_dissector/
    (registered by a new late_initcall in net/core/flow_dissector.c
    via register_net_sysctl(&init_net, "net/flow_dissector", tbl) —
    same shape as net/mpls, net/bridge).

  - Forward declarations. flow_dissect_fast_ipv4 / _ipv6 are forward-
    declared at the top of the file (after includes). Subsequent
    patches add flow_dissect_fast_vlan etc. that call these helpers;
    anchoring the declarations at top of file makes the patch insertion
    order robust against kernel-version line-number drift.

  - Cost model. On a miss, dispatcher overhead is one ethertype switch
    (~5 cycles) plus the per-case static_branch test (~not-taken JMP
    when 0). On a hit, the straight-line body saves ~30 ns/skb of
    graph-walk cost on the common shape (Zen 2 / Cortex-A76; see the
    cover letter for the cross-uarch microbench table).

  - Output contract. Byte-identical with the slow path for the
    eligible shape. The IPv6 fast-path defers on any non-zero flow
    label: callers passing FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL --
    skb_get_hash(), i.e. RPS/RFS, fq, fq_codel, cake -- make the slow
    path stop at a non-zero label, and dissectors requesting the label
    key write it; deferring on the label itself keeps byte-identity
    for every combination without threading flags into the helpers,
    and zero-label IPv6 stays on the fast path. Fragmented IPv4
    packets defer. Custom dissectors defer, as do FLOW_DISSECTOR_F_*
    flags other than PARSE_1ST_FRAG and STOP_AT_FLOW_LABEL. Admitting
    STOP_AT_FLOW_LABEL is what makes the fast path reachable from
    skb_get_hash() at all -- the kernel's main dissection consumer.

Assisted-by: Claude:claude-fable-5 sparse smatch
Signed-off-by: Dave Seddon <dave.seddon.ca@xxxxxxxxx>
---
 Documentation/admin-guide/sysctl/net.rst |  27 ++-
 include/net/flow_dissector.h             |   5 +
 net/core/flow_dissector.c                | 273 ++++++++++++++++++++++-
 3 files changed, 302 insertions(+), 3 deletions(-)

diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin-guide/sysctl/net.rst
index e586e17fc7a5..aac82169e019 100644
--- a/Documentation/admin-guide/sysctl/net.rst
+++ b/Documentation/admin-guide/sysctl/net.rst
@@ -460,7 +460,32 @@ historical importance.
 
 Default: 0
 
-2. /proc/sys/net/unix - Parameters for Unix domain sockets
+2. /proc/sys/net/flow_dissector - Flow dissector fast-path per-shape sysctls
+---------------------------------------------------------------------------
+
+The flow dissector is invoked on every packet that needs a flow hash
+(RPS, RFS, ECMP, sch_cake, cls_flower, ...). For common shapes
+(Eth+IPv4/IPv6+TCP/UDP, plus per-shape extensions for VLAN/QinQ,
+PPPoE, MPLS, IP-in-IP and GRE added by this series), an opt-in
+fast-path bypasses the slow-path graph walk and writes
+``struct flow_keys`` directly. Each shape is gated by its own static
+key + sysctl so operators enable only what their deployment uses.
+
+All defaults are 0 (off). Flipping any of these on takes effect
+immediately; no kernel rebuild needed.
+
+eth_ip
+~~~~~~
+
+Eth + IPv4 (IHL=5, no fragmentation) + TCP/UDP, and Eth + IPv6 (no
+extension headers) + TCP/UDP. Output is byte-identical to the slow
+path on the eligible shape; non-matching packets fall through to the
+slow path unchanged. The dispatcher costs one not-taken JMP per call
+when this is 0.
+
+Default: 0
+
+3. /proc/sys/net/unix - Parameters for Unix domain sockets
 ----------------------------------------------------------
 
 There is only one file in this directory.
diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h
index ced79dc8e856..7d513a6cbb55 100644
--- a/include/net/flow_dissector.h
+++ b/include/net/flow_dissector.h
@@ -425,6 +425,11 @@ __be32 flow_get_u32_dst(const struct flow_keys *flow);
 extern struct flow_dissector flow_keys_dissector;
 extern struct flow_dissector flow_keys_basic_dissector;
 
+/* Per-shape fast-path static-branch keys, toggled via
+ * /proc/sys/net/flow_dissector/<shape>. All default off.
+ */
+extern struct static_key_false flow_dissector_eth_ip_key;
+
 /* struct flow_keys_digest:
  *
  * This structure is used to hold a digest of the full flow keys. This is a
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index b8c59209b460..88b1bd1cc4ca 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -37,6 +37,33 @@
 #endif
 #include <linux/bpf-netns.h>
 
+/* Per-shape fast-path gates, one static_branch per shape so operators
+ * enable only what their deployment carries. All default off.
+ */
+DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key);
+EXPORT_SYMBOL(flow_dissector_eth_ip_key);
+
+/* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */
+#define FLOW_DIS_IPV4_VIHL_NOOPT	0x45
+
+/* Fast-path helper forward declarations. */
+static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen);
+static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen);
+
+/* One of the two dissectors the fast-path eligibility check admits;
+ * defined here so flow_dissect_fast() below can reference it (its keys
+ * and init live near the bottom of the file).
+ */
+static struct flow_dissector flow_keys_dissector_symmetric __read_mostly;
+
 static void dissector_set_key(struct flow_dissector *flow_dissector,
 			      enum flow_dissector_key_id key_id)
 {
@@ -1043,6 +1070,223 @@ static bool is_pppoe_ses_hdr_valid(const struct pppoe_hdr *hdr)
 	return hdr->ver == 1 && hdr->type == 1 && hdr->code == 0;
 }
 
+/* The IPv4 fast-path assumes a 20-byte IPv4 header (IHL == 5). The
+ * runtime IHL check below is the dynamic invariant; this static
+ * assertion is the compile-time one, in the unlikely case struct
+ * iphdr ever grows.
+ */
+static_assert(sizeof(struct iphdr) == 20);
+
+/* Straight-line eth + IPv4 (IHL=5, not fragmented) + TCP|UDP. Returns
+ * true when @target_container has been filled byte-identically to the
+ * slow path; false on any miss (caller falls back to the slow path).
+ */
+static bool flow_dissect_fast_ipv4(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	struct flow_dissector_key_addrs *key_addrs;
+	struct flow_dissector_key_basic *key_basic;
+	struct flow_dissector_key_ports *key_ports;
+	const struct iphdr *iph;
+	int thoff;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4))
+		return false;
+
+	iph = (const struct iphdr *)((const u8 *)data + nhoff);
+
+	if (unlikely(*(const u8 *)iph != FLOW_DIS_IPV4_VIHL_NOOPT))
+		return false;
+
+	if (unlikely(iph->frag_off & htons(IP_MF | IP_OFFSET)))
+		return false;
+
+	if (unlikely(iph->protocol != IPPROTO_TCP &&
+		     iph->protocol != IPPROTO_UDP))
+		return false;
+
+	thoff = nhoff + (int)sizeof(*iph);
+
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->addr_type = FLOW_DISSECTOR_KEY_IPV4_ADDRS;
+		key_control->thoff = min_t(u16, thoff,
+					   skb ? skb->len : hlen);
+		key_control->flags = 0;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_BASIC)) {
+		key_basic = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_BASIC,
+						      target_container);
+		key_basic->n_proto = htons(ETH_P_IP);
+		key_basic->ip_proto = iph->protocol;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_IPV4_ADDRS)) {
+		key_addrs = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_IPV4_ADDRS,
+						      target_container);
+		memcpy(&key_addrs->v4addrs.src, &iph->saddr,
+		       sizeof(key_addrs->v4addrs.src));
+		memcpy(&key_addrs->v4addrs.dst, &iph->daddr,
+		       sizeof(key_addrs->v4addrs.dst));
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_PORTS)) {
+		const __be32 *ports = (const __be32 *)
+			((const u8 *)data + thoff);
+
+		key_ports = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_PORTS,
+						      target_container);
+		key_ports->ports = *ports;
+	}
+
+	return true;
+}
+
+/* Same as flow_dissect_fast_ipv4 but for a fixed 40-byte IPv6 header
+ * (no extension headers). The nexthdr check below holds the run-time
+ * invariant; the static_assert holds the compile-time one.
+ */
+static_assert(sizeof(struct ipv6hdr) == 40);
+
+static bool flow_dissect_fast_ipv6(const struct sk_buff *skb,
+				   struct flow_dissector *flow_dissector,
+				   void *target_container,
+				   const void *data,
+				   int nhoff, int hlen)
+{
+	struct flow_dissector_key_control *key_control;
+	struct flow_dissector_key_addrs *key_addrs;
+	struct flow_dissector_key_basic *key_basic;
+	struct flow_dissector_key_ports *key_ports;
+	const struct ipv6hdr *iph;
+	int thoff;
+
+	if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4))
+		return false;
+
+	iph = (const struct ipv6hdr *)((const u8 *)data + nhoff);
+
+	if (unlikely((*(const u8 *)iph >> 4) != 6))
+		return false;
+
+	/* Any non-zero flow label defers, checked before everything else
+	 * (including the tunnel descents below). Callers passing
+	 * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL (skb_get_hash() -- so
+	 * RPS/RFS, fq, cake) make the slow path stop at a non-zero label
+	 * even on a tunnel outer, and dissectors requesting the label key
+	 * write it; both diverge from the plain parse. Deferring on the
+	 * label itself covers every combination without threading flags
+	 * into the helpers, and zero-label IPv6 (the overwhelmingly
+	 * common case) stays on the fast path.
+	 */
+	if (unlikely((iph->flow_lbl[0] & 0x0f) |
+		     iph->flow_lbl[1] | iph->flow_lbl[2]))
+		return false;
+
+	if (unlikely(iph->nexthdr != IPPROTO_TCP &&
+		     iph->nexthdr != IPPROTO_UDP))
+		return false;
+
+	thoff = nhoff + (int)sizeof(*iph);
+
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_CONTROL)) {
+		key_control = skb_flow_dissector_target(flow_dissector,
+							FLOW_DISSECTOR_KEY_CONTROL,
+							target_container);
+		key_control->addr_type = FLOW_DISSECTOR_KEY_IPV6_ADDRS;
+		key_control->thoff = min_t(u16, thoff,
+					   skb ? skb->len : hlen);
+		key_control->flags = 0;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_BASIC)) {
+		key_basic = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_BASIC,
+						      target_container);
+		key_basic->n_proto = htons(ETH_P_IPV6);
+		key_basic->ip_proto = iph->nexthdr;
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_IPV6_ADDRS)) {
+		key_addrs = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_IPV6_ADDRS,
+						      target_container);
+		memcpy(&key_addrs->v6addrs.src, &iph->saddr,
+		       sizeof(key_addrs->v6addrs.src));
+		memcpy(&key_addrs->v6addrs.dst, &iph->daddr,
+		       sizeof(key_addrs->v6addrs.dst));
+	}
+	if (dissector_uses_key(flow_dissector,
+			       FLOW_DISSECTOR_KEY_PORTS)) {
+		const __be32 *ports = (const __be32 *)
+			((const u8 *)data + thoff);
+
+		key_ports = skb_flow_dissector_target(flow_dissector,
+						      FLOW_DISSECTOR_KEY_PORTS,
+						      target_container);
+		key_ports->ports = *ports;
+	}
+
+	return true;
+}
+
+/* Top-level dispatcher: eligibility check (only the two standard
+ * dissectors and flag subset) + per-proto switch with per-shape
+ * static_branch gating. Each case's branch is a forward not-taken JMP
+ * when its sysctl is 0 — matches the slow-path cost.
+ */
+static bool flow_dissect_fast(const struct sk_buff *skb,
+			      struct flow_dissector *flow_dissector,
+			      void *target_container,
+			      const void *data,
+			      __be16 proto, int nhoff, int hlen,
+			      unsigned int flags)
+{
+	if (flow_dissector != &flow_keys_dissector &&
+	    flow_dissector != &flow_keys_dissector_symmetric)
+		return false;
+
+	/* skb_get_hash() -- behind RPS/RFS, fq, fq_codel, cake -- passes
+	 * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL on every dissect, so the
+	 * flag must be admitted or the fast path never runs for the
+	 * kernel's main consumers. Its only semantic (stop early on a
+	 * non-zero IPv6 flow label) is subsumed by the IPv6 fast path
+	 * deferring on any non-zero label. Any other flag defers.
+	 */
+	if (flags & ~(unsigned int)(FLOW_DISSECTOR_F_PARSE_1ST_FRAG |
+				    FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL))
+		return false;
+
+	switch (proto) {
+	case htons(ETH_P_IP):
+		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
+			return false;
+		return flow_dissect_fast_ipv4(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	case htons(ETH_P_IPV6):
+		if (!static_branch_unlikely(&flow_dissector_eth_ip_key))
+			return false;
+		return flow_dissect_fast_ipv6(skb, flow_dissector,
+					      target_container, data,
+					      nhoff, hlen);
+	default:
+		return false;
+	}
+}
+
 /**
  * __skb_flow_dissect - extract the flow_keys struct and return it
  * @net: associated network namespace, derived from @skb if NULL
@@ -1183,6 +1427,13 @@ bool __skb_flow_dissect(const struct net *net,
 		rcu_read_unlock();
 	}
 
+	/* Opt-in fast-path; flow_dissect_fast() does the eligibility check
+	 * and per-shape gating.
+	 */
+	if (flow_dissect_fast(skb, flow_dissector, target_container,
+			      data, proto, nhoff, hlen, flags))
+		return true;
+
 	if (dissector_uses_key(flow_dissector,
 			       FLOW_DISSECTOR_KEY_ETH_ADDRS)) {
 		struct flow_dissector_key_eth_addrs *key_eth_addrs;
@@ -1879,8 +2130,6 @@ void make_flow_keys_digest(struct flow_keys_digest *digest,
 }
 EXPORT_SYMBOL(make_flow_keys_digest);
 
-static struct flow_dissector flow_keys_dissector_symmetric __read_mostly;
-
 u32 __skb_get_hash_symmetric_net(const struct net *net, const struct sk_buff *skb)
 {
 	struct flow_keys keys;
@@ -2116,3 +2365,23 @@ static int __init init_default_flow_dissectors(void)
 	return 0;
 }
 core_initcall(init_default_flow_dissectors);
+
+/* Per-shape fast-path sysctls under /proc/sys/net/flow_dissector/. */
+static struct ctl_table flow_dissector_sysctl_table[] = {
+	{
+		.procname	= "eth_ip",
+		.data		= &flow_dissector_eth_ip_key.key,
+		.maxlen		= sizeof(flow_dissector_eth_ip_key),
+		.mode		= 0644,
+		.proc_handler	= proc_do_static_key,
+	},
+};
+
+static int __init flow_dissector_sysctl_init(void)
+{
+	if (!register_net_sysctl(&init_net, "net/flow_dissector",
+				 flow_dissector_sysctl_table))
+		return -ENOMEM;
+	return 0;
+}
+late_initcall(flow_dissector_sysctl_init);
-- 
2.54.0