net: flow_offload: existing_qdiscs_register() splices into live container cb_lists without holding the container lock (list corruption / use-after-free)
From: 徐晗翔
Date: Wed Oct 28 2026 - 22:46:37 EST
When a flow_indr driver registers, existing_qdiscs_register()
(net/core/flow_offload.c:415-428) replays the bind for every already
bound container by splicing the new flow_block_cb list directly into
the container's live cb_list (tc block / nftables base chain / nf
flowtable) -- holding only the framework mutex flow_indr_block_lock,
never the container lock that maintains that list's invariant -- so
the splice's four unsynchronized pointer writes race any concurrent
traversal, bind or unbind of the same list, with no common lock on any
of the three variants (the nftables variant has no rtnl anywhere on
either side). I verified the lock-domain bypass deterministically on
a pristine Linux 7.3 KASAN kernel, and verified the resulting damage
in the concurrent-registration shape (two drivers probing at once,
e.g. ice + sfc): 440 list_del corruption reports and 879
slab-use-after-free reports in one run, exactly the
lost/double-linkage -> double-processing -> UAF mode predicted from
the code.
The registration contexts are provably not serialized against the
container side: ice (ice_init_pf_sw from PCI probe / devlink reload),
sfc (ef100 probe), mlx5e (auxiliary bus probe) and nfp register
without rtnl; tc readers under cb_lock may additionally run without
rtnl for unlocked_driver_cb blocks; and the nftables commit path
holds only commit_mutex from the netlink entry down to the cb_list
traversal -- so for the nft variant there is no common lock with the
splicing side regardless of rtnl. The triggering pair is an
admin-side event (driver probe, PCI hotplug, devlink eswitch mode
switch) overlapping ordinary tc/nft/flowtable operations
(CAP_NET_ADMIN in a netns suffices). Fix direction: the replay splice
must happen under the container's own lock (record the container type
in flow_indr_dev_info and take cb_lock/commit_mutex/
flow_block_lock around the splice, with a defined ordering against
flow_indr_block_lock), or the splice decision must move to the
container side.
The detailed description:
* Test environment / affected version
Linux version 7.3.0 (gcc 13.3.0, GNU ld 2.42) #1 SMP PREEMPT_DYNAMIC
-- vanilla mainline 7.3 release tarball, KASAN +
CONFIG_DEBUG_LIST=y + CONFIG_LIST_HARDENED=y, QEMU/KVM x86_64 (8
vCPU). Framework code exercised (flow_indr_dev_register,
existing_qdiscs_register, flow_indr_block_cb_alloc, flow_block_cb_*)
is unmodified in-tree net/core. Fidelity note: the container side
(a tc-style block with rwsem cb_lock and the cls_api.c
bind/unbind/cleanup/reader sequences) is replicated in the attached
module, mirroring net/sched/cls_api.c line-by-line, because the
test VM has no switchdev NIC; the raced list, the block_cb objects
and the whole registration/replay path are native.
* The code
net/core/flow_offload.c:415-428:
static void existing_qdiscs_register(...)
{
struct flow_block_offload bo; /* stack */
struct flow_indir_dev_info *cur;
list_for_each_entry(cur, &flow_indr_dev_list, list) {
memset(&bo, 0, sizeof(bo));
...
INIT_LIST_HEAD(&bo.cb_list);
cb(cur->dev, cur->sch, cb_priv, cur->type,
&bo, cur->data, cur->cleanup);
list_splice(&bo.cb_list, cur->cb_list); /* :426 */
}
}
cur->cb_list points at the live list head of one of the three
containers (tc: &block->flow_block.cb_list, cls_api.c:805; nft:
&basechain->flow_block.cb_list, nf_tables_offload.c:383; ft:
&flowtable->flow_block.cb_list, nf_flow_table_offload.c:1222).
Every other producer/consumer of those lists runs under the
container lock; this splice -- reachable whenever a second driver
registers while containers are bound -- is the only unlocked
producer. list_splice() performs four pointer writes
(include/linux/list.h:570-594): any interleaved traversal or
list_del observes a half-spliced list.
* Failure modes (from the code; the second is the dynamically
confirmed one)
(i) torn read: a concurrent list_for_each_entry reader can see the
new head->next while last->next still points into the
registerer's stack bo.cb_list -- it then treats stack memory
as a struct flow_block_cb and calls ->cb (type confusion /
near-arbitrary call);
(ii) lost/double linkage vs a concurrent locked splice/unbind:
nodes survive but fall out of (or land in two) list views;
later free-walks double-process them -- the observed
list_del corruption and slab-UAF in the unregister cleanup
walk;
(iii) torn writes against a locked list_del of the same node.
* Evidence (pristine 7.3; module attached)
- Deterministic lock bypass (action=casea_ctl): while the
container's cb_lock is exclusively held (down_write, standing in
for any container-side critical section), a
flow_indr_dev_register(D2) replay lands its new block_cb on the
live cb_list -- splice_landed_while_exclusive_lock_held=1,
d2_cb_on_list=1. By construction, no container lock is taken by
the registration path.
- Concurrent-registration natural run (action=casea_nat2; the
real-world shape is two drivers probing concurrently, e.g. ice +
sfc hotplug): two unlocked splices racing on the same list head,
2x10^5 register/unregister cycles each -> 440 "list_del
corruption. next->prev should be X, but was Y" reports and 879
slab-use-after-free reports, all with the stack
__list_del_entry_valid_or_report <- tc-style cleanup <-
flow_indr_dev_unregister -- the orphaned/double-linked block_cbs
being freed through one list view and deleted through the other.
- Single-driver vs reader (action=casea_nat): 20,000 registration
cycles x 3.46M reader traversals produced zero corruption -- the
torn-read window (mode (i)) is nanosecond-scale on both sides and
needs ~10^6+ cycles; the concurrent-registration shape (mode
(ii)) corrupts within thousands. I report both numbers so the
reachability statement is precise: the protection gap is
structural; the damage is most practically reached when two
registrations overlap.
- Baseline (flowsanity): full register -> bind -> read -> unbind ->
unregister sequence with no overlap: zero splats -- the module
and replicated container are clean when not interleaved.
Module usage (attached flowrace.c; build as external module against
a DEBUG_LIST/LIST_HARDENED kernel):
insmod flowrace.ko action=casea_ctl
insmod flowrace.ko action=casea_nat2 # corruption storm;
# timeout exit expected
Regards,
Hanxiang Xu
xuhx@xxxxxxxxxxx
// SPDX-License-Identifier: GPL-2.0
/*
* flowrace.c - dynamic verification for the net/core audit report
* docs/manual-review-net-core-2026-10-28.md (4th round, submission version).
*
* Verified cases (kernel source untouched; only EXPORT_SYMBOL APIs):
* Case A: flow_indr_dev_register()/existing_qdiscs_register() splices new
* block_cbs into a live container's cb_list while holding only
* flow_indr_block_lock, bypassing the container lock
* (action=casea_ctl deterministic lock-bypass proof, casea_nat
* natural corruption hammer with concurrent cb_list readers).
* Case B: flow_indr_dev_unregister() collection/notify window - container
* teardown during the window leaves the block_cb attached, the
* container is freed, then the notify cleanup callback touches the
* freed container (action=caseb, deterministic: cleanup callback of
* container #1 performs container #2's teardown+free mid-notify).
* Case C: shared percpu metadata_dst scratch (xfrm_bpf_md_dst pattern) -
* bpf_skb_set_xfrm_info()/xfrmi_xmit2() replica on real in-tree
* objects (metadata_dst_alloc_percpu + dst_alloc + real skbs):
* leak variant, double-consume variant (deterministic accounting
* proof + KASAN UAF), natural two-thread same-CPU churn
* (action=casec_ctl / casec_nat).
*
* Fidelity notes (see results doc): the framework code paths
* (flow_indr_dev_register/unregister/setup_offload, flow_indr_block_cb_alloc,
* flow_block_cb_*, metadata_dst_*, dst_*) all run unmodified from net/core;
* the module only replicates the in-tree *call sites* (tc container side,
* driver indr callback side, bpf kfunc / xfrmi xmit sequences) that live in
* cls_api.c / ice_main.c / xfrm_interface_bpf.c / xfrm_interface_core.c.
*/
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/slab.h>
#include <linux/kthread.h>
#include <linux/sched.h>
#include <linux/skbuff.h>
#include <linux/netdevice.h>
#include <linux/pkt_cls.h>
#include <linux/delay.h>
#include <linux/wait.h>
#include <linux/random.h>
#include <linux/completion.h>
#include <net/net_namespace.h>
#include <net/flow_offload.h>
#include <net/dst.h>
#include <net/dst_metadata.h>
#define FRTAG "FLOWRACE"
static char *action = "sanity";
module_param(action, charp, 0444);
MODULE_PARM_DESC(action, "sanity|casea_ctl|casea_nat|casea_nat2|caseb|casec_ctl|casec_nat");
static int iters = 2000;
module_param(iters, int, 0444);
static int nreaders = 4;
module_param(nreaders, int, 0444);
static int jitter_us = 40;
module_param(jitter_us, int, 0444);
/* ------------------------------------------------------------------ */
/* shared state */
struct fake_block { /* replica of tcf_block's offload parts */
struct flow_block fb; /* fb.cb_list == block->flow_block.cb_list */
struct rw_semaphore cb_lock;/* replica of tcf_block.cb_lock */
};
static struct net_device *fr_ndev; /* any netdev; passed through as indr.dev */
static int fr_d1, fr_d2, fr_d3; /* registration privs (drivers 1/2/3) */
static LIST_HEAD(fr_d1_block_list); /* driver_list replicas */
static atomic_t fr_cb_calls; /* reader invocations of block_cb->cb */
static struct completion fr_cc_done; /* bounded worker completion */
static atomic_t fr_caseb_early_ret; /* teardown unbind returned -EOPNOTSUPP */
static int fr_caseb_phase; /* caseb: cleanup(cb1) performs fb2 td */
static struct fake_block *fr_caseb_fb1, *fr_caseb_fb2;
/* ------------------------------------------------------------------ */
/* Case C state */
static struct kmem_cache *fr_dst_cache;
static struct dst_ops fr_dst_ops = { .family = AF_INET };
static struct metadata_dst __percpu *fr_scratch; /* the shared percpu md */
static atomic_t fr_dst_created;
static atomic_t fr_dst_destroyed;
static void fr_dst_destroy(struct dst_entry *dst)
{
atomic_inc(&fr_dst_destroyed);
}
/* ------------------------------------------------------------------ */
/* driver-side indr callback: replica of ice_indr_setup_tc_block() */
/* (ice_main.c:9502-9580) / mlx5e_rep_indr_setup_block() (rep/tc.c) */
static int fr_block_cb(enum tc_setup_type type, void *type_data, void *cb_priv)
{
atomic_inc(&fr_cb_calls);
return 0;
}
static void fr_cb_release(void *cb_priv) { }
static int fr_indr_bind_cb(struct net_device *dev, struct Qdisc *sch,
void *cb_priv, enum tc_setup_type type,
void *type_data, void *data,
void (*cleanup)(struct flow_block_cb *block_cb))
{
struct flow_block_offload *f = type_data;
struct flow_block_cb *block_cb;
if (type != TC_SETUP_BLOCK || !data)
return -EOPNOTSUPP;
switch (f->command) {
case FLOW_BLOCK_BIND:
block_cb = flow_indr_block_cb_alloc(fr_block_cb, cb_priv,
cb_priv, fr_cb_release,
f, dev, sch, data,
cb_priv, cleanup);
if (IS_ERR(block_cb))
return PTR_ERR(block_cb);
flow_block_cb_add(block_cb, f);
list_add_tail(&block_cb->driver_list, &fr_d1_block_list);
return 0;
case FLOW_BLOCK_UNBIND:
block_cb = flow_block_cb_lookup(f->block, fr_block_cb, cb_priv);
if (!block_cb)
return -ENOENT;
flow_indr_block_cb_remove(block_cb, f);
list_del(&block_cb->driver_list);
return 0;
default:
return -EOPNOTSUPP;
}
}
/* ------------------------------------------------------------------ */
/* container-side replicas of the tc paths (cls_api.c) */
/* replica of tcf_block_offload_init() (cls_api.c:791-807) */
static void fr_bo_init(struct flow_block_offload *bo, struct fake_block *fb,
enum flow_block_command cmd)
{
memset(bo, 0, sizeof(*bo));
bo->command = cmd;
bo->binder_type = FLOW_BLOCK_BINDER_TYPE_CLSACT_INGRESS;
bo->block = &fb->fb;
bo->cb_list_head = &fb->fb.cb_list; /* indir_dev_add records this */
INIT_LIST_HEAD(&bo->cb_list);
}
/* replica of tcf_block_unbind() tail (cls_api.c:1685-1699) */
static void fr_unbind_list(struct flow_block_offload *bo)
{
struct flow_block_cb *block_cb, *n;
list_for_each_entry_safe(block_cb, n, &bo->cb_list, list) {
list_del(&block_cb->list);
flow_block_cb_free(block_cb);
}
}
/* replica of tc_block_indr_cleanup() (cls_api.c:812-831) */
static void fr_tc_cleanup(struct flow_block_cb *block_cb)
{
struct fake_block *fb = block_cb->indr.data;
struct flow_block_offload bo;
fr_bo_init(&bo, fb, FLOW_BLOCK_UNBIND);
/* NOTE: no reference to fb, no RCU read section - exactly like the
* in-tree callback: if fb was freed during the unregister window,
* every access below is use-after-free. */
down_write(&fb->cb_lock);
list_del(&block_cb->driver_list);
list_move(&block_cb->list, &bo.cb_list);
fr_unbind_list(&bo);
up_write(&fb->cb_lock);
/* caseb choreography: while notify() is between container #1 and
* container #2, perform container #2's teardown exactly like a
* concurrent qdisc destroy would: driver absent -> unbind is a
* no-op leaving the cb attached, then the container is freed
* (models tcf_block_destroy() after the RCU grace period). */
if (fr_caseb_phase && fb == fr_caseb_fb1 && fr_caseb_fb2) {
struct fake_block *fb2 = fr_caseb_fb2;
struct flow_block_offload bo2;
int err;
fr_bo_init(&bo2, fb2, FLOW_BLOCK_UNBIND);
err = flow_indr_dev_setup_offload(fr_ndev, NULL,
TC_SETUP_BLOCK, fb2, &bo2,
fr_tc_cleanup);
if (err == -EOPNOTSUPP && list_empty(&bo2.cb_list))
atomic_set(&fr_caseb_early_ret, 1);
kfree(fb2); /* container destroyed, cb attached */
fr_caseb_fb2 = NULL;
pr_info(FRTAG " caseb: fb2 teardown+free done mid-notify (early_ret=%d)\n",
atomic_read(&fr_caseb_early_ret));
}
}
/* replica of tcf_block_offload_bind() (cls_api.c:870-908 incl. cmd+splice) */
static int fr_container_bind(struct fake_block *fb)
{
struct flow_block_offload bo;
int err;
fr_bo_init(&bo, fb, FLOW_BLOCK_BIND);
down_write(&fb->cb_lock);
err = flow_indr_dev_setup_offload(fr_ndev, NULL, TC_SETUP_BLOCK, fb,
&bo, fr_tc_cleanup);
/* in-tree indr path ignores err here and splices unconditionally */
list_splice(&bo.cb_list, &fb->fb.cb_list);
up_write(&fb->cb_lock);
return err;
}
/* replica of tcf_block_offload_unbind() (cls_api.c:910-926) */
static void fr_container_unbind(struct fake_block *fb)
{
struct flow_block_offload bo;
fr_bo_init(&bo, fb, FLOW_BLOCK_UNBIND);
down_write(&fb->cb_lock);
flow_indr_dev_setup_offload(fr_ndev, NULL, TC_SETUP_BLOCK, fb, &bo,
fr_tc_cleanup);
fr_unbind_list(&bo);
up_write(&fb->cb_lock);
}
/* replica of tc_setup_cb_call()/__tc_setup_cb_call() (cls_api.c:3638-3682) */
static int fr_reader_pass(struct fake_block *fb)
{
struct flow_block_cb *block_cb;
int cnt = 0;
down_read(&fb->cb_lock);
list_for_each_entry(block_cb, &fb->fb.cb_list, list) {
block_cb->cb(TC_SETUP_CLSFLOWER, NULL, block_cb->cb_priv);
cnt++;
}
up_read(&fb->cb_lock);
return cnt;
}
static struct fake_block *fr_block_alloc(void)
{
struct fake_block *fb = kzalloc(sizeof(*fb), GFP_KERNEL);
if (!fb)
return NULL;
INIT_LIST_HEAD(&fb->fb.cb_list);
init_rwsem(&fb->cb_lock);
return fb;
}
/* ------------------------------------------------------------------ */
/* action: sanity */
static int fr_do_sanity(void)
{
struct fake_block *fb = fr_block_alloc();
int cnt, err;
if (!fb)
return -ENOMEM;
err = flow_indr_dev_register(fr_indr_bind_cb, &fr_d1);
pr_info(FRTAG " sanity: register D1 err=%d\n", err);
err = fr_container_bind(fb);
cnt = fr_reader_pass(fb);
pr_info(FRTAG " sanity: bind err=%d reader entries=%d\n", err, cnt);
fr_container_unbind(fb);
cnt = fr_reader_pass(fb);
pr_info(FRTAG " sanity: after unbind entries=%d (expect 0)\n", cnt);
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d1, fr_cb_release);
pr_info(FRTAG " sanity: unregister D1 done, cb_calls=%d\n",
atomic_read(&fr_cb_calls));
kfree(fb);
return 0;
}
/* ------------------------------------------------------------------ */
/* action: casea_ctl - deterministic container-lock bypass proof */
static atomic_t holder_in, reg_done, holder_found;
static int fr_holder_fn(void *data)
{
struct fake_block *fb = data;
struct flow_block_cb *block_cb;
int found = 0;
down_write(&fb->cb_lock); /* exclusive: container locked */
atomic_set(&holder_in, 1);
while (!atomic_read(®_done))
msleep(5);
/* we still hold the exclusive lock: if D2's block_cb is on the list
* now, the register-path splice bypassed cb_lock by construction */
list_for_each_entry(block_cb, &fb->fb.cb_list, list)
if (block_cb->cb_ident == &fr_d2)
found = 1;
atomic_set(&holder_found, found);
up_write(&fb->cb_lock);
return 0;
}
static int fr_do_casea_ctl(void)
{
struct fake_block *fb = fr_block_alloc();
struct task_struct *holder;
int err;
if (!fb)
return -ENOMEM;
flow_indr_dev_register(fr_indr_bind_cb, &fr_d1);
fr_container_bind(fb);
holder = kthread_run(fr_holder_fn, fb, "fr_holder");
while (!atomic_read(&holder_in))
msleep(5);
/* D2 registers while the container lock is held exclusively by the
* holder thread: existing_qdiscs_register() -> list_splice() runs
* with no cb_lock (ice/mlx5e probe context in the real system). */
err = flow_indr_dev_register(fr_indr_bind_cb, &fr_d2);
atomic_set(®_done, 1);
kthread_stop(holder);
pr_info(FRTAG " CASEA-CTL: register_D2 err=%d splice_landed_while_exclusive_lock_held=%d d2_cb_on_list=%d\n",
err, atomic_read(&holder_found), atomic_read(&holder_found));
/* clean teardown: container still alive, both drivers' cbs removed
* through their proper paths; zero KASAN expected for this plan */
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d2, fr_cb_release);
fr_container_unbind(fb);
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d1, fr_cb_release);
pr_info(FRTAG " CASEA-CTL: teardown done, list_empty=%d\n",
list_empty(&fb->fb.cb_list));
kfree(fb);
return 0;
}
/* ------------------------------------------------------------------ */
/* action: casea_nat - natural corruption hammer */
static atomic_t fr_stop;
static atomic_t fr_passes, fr_entries, fr_cycles;
static atomic_t fr_readers_alive;
static int fr_reader_fn(void *data)
{
struct fake_block *fb = data;
atomic_inc(&fr_readers_alive);
while (!atomic_read(&fr_stop)) {
int n = fr_reader_pass(fb);
atomic_inc(&fr_passes);
atomic_add(n, &fr_entries);
cond_resched();
}
atomic_dec(&fr_readers_alive);
return 0;
}
static int fr_do_casea_nat(void)
{
struct fake_block *fb = fr_block_alloc();
struct task_struct **rd;
int i, err;
if (!fb)
return -ENOMEM;
atomic_set(&fr_stop, 0);
flow_indr_dev_register(fr_indr_bind_cb, &fr_d1);
fr_container_bind(fb);
rd = kcalloc(nreaders, sizeof(*rd), GFP_KERNEL);
for (i = 0; i < nreaders; i++)
rd[i] = kthread_run(fr_reader_fn, fb, "fr_rd%d", i);
/* churn: each D2 register performs one lockless splice into the live
* cb_list (existing_qdiscs_register), racing the locked readers */
for (i = 0; i < iters; i++) {
err = flow_indr_dev_register(fr_indr_bind_cb, &fr_d2);
if (!err)
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d2,
fr_cb_release);
atomic_inc(&fr_cycles);
if (!(i % 256))
cond_resched();
}
atomic_set(&fr_stop, 1);
for (i = 0; i < nreaders; i++)
if (rd[i])
kthread_stop(rd[i]);
kfree(rd);
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d1, fr_cb_release);
pr_info(FRTAG " CASEA-NAT: cycles=%d reader_passes=%d entries=%d cb_calls=%d readers_alive=%d\n",
atomic_read(&fr_cycles), atomic_read(&fr_passes),
atomic_read(&fr_entries), atomic_read(&fr_cb_calls),
atomic_read(&fr_readers_alive));
kfree(fb);
return 0;
}
/* concurrent-driver variant: two drivers register at the same time (as
* with ice+sfc hotplug probing in parallel) - two lockless splices race
* on the same list head; an interleave orphans or double-links a chain
* and LIST_HARDENED fires on the next list_del. */
static int fr_churn_fn(void *priv)
{
int i;
for (i = 0; i < iters; i++) {
if (!flow_indr_dev_register(fr_indr_bind_cb, priv))
flow_indr_dev_unregister(fr_indr_bind_cb, priv,
fr_cb_release);
atomic_inc(&fr_cycles);
cond_resched();
}
complete(&fr_cc_done);
return 0;
}
static int fr_do_casea_nat2(void)
{
struct fake_block *fb = fr_block_alloc();
struct task_struct *c2, *c3, *r0, *r1;
if (!fb)
return -ENOMEM;
atomic_set(&fr_stop, 0);
flow_indr_dev_register(fr_indr_bind_cb, &fr_d1);
fr_container_bind(fb);
init_completion(&fr_cc_done);
r0 = kthread_run(fr_reader_fn, fb, "fr_rd0");
r1 = kthread_run(fr_reader_fn, fb, "fr_rd1");
c2 = kthread_run(fr_churn_fn, &fr_d2, "fr_ch2");
c3 = kthread_run(fr_churn_fn, &fr_d3, "fr_ch3");
wait_for_completion(&fr_cc_done);
wait_for_completion(&fr_cc_done);
atomic_set(&fr_stop, 1);
kthread_stop(r0);
kthread_stop(r1);
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d1, fr_cb_release);
pr_info(FRTAG " CASEA-NAT2: cycles=%d reader_passes=%d entries=%d cb_calls=%d\n",
atomic_read(&fr_cycles), atomic_read(&fr_passes),
atomic_read(&fr_entries), atomic_read(&fr_cb_calls));
kfree(fb);
return 0;
}
/* ------------------------------------------------------------------ */
/* action: caseb - deterministic unregister-window UAF */
static int fr_do_caseb(void)
{
struct fake_block *fb1, *fb2;
fb1 = fr_block_alloc();
fb2 = fr_block_alloc();
if (!fb1 || !fb2)
return -ENOMEM;
flow_indr_dev_register(fr_indr_bind_cb, &fr_d1);
fr_container_bind(fb1);
fr_container_bind(fb2);
pr_info(FRTAG " caseb: two containers bound\n");
/* unregister: mutex section removes D1 + collects both cbs; the
* notify phase then calls fr_tc_cleanup(cb_fb1) first - which we
* use as the controlled point to run container #2's teardown+free
* (equivalent to a concurrent qdisc destroy inside the window),
* after which notify calls fr_tc_cleanup(cb_fb2) on freed fb2. */
fr_caseb_fb1 = fb1;
fr_caseb_fb2 = fb2;
fr_caseb_phase = 1;
flow_indr_dev_unregister(fr_indr_bind_cb, &fr_d1, fr_cb_release);
fr_caseb_phase = 0;
pr_info(FRTAG " CASEB: unregister done, early_ret=%d fb1_list_empty=%d (fb2 kfree'd mid-notify; UAF expected in dmesg)\n",
atomic_read(&fr_caseb_early_ret), list_empty(&fb1->fb.cb_list));
kfree(fb1);
return 0;
}
/* ------------------------------------------------------------------ */
/* Case C: percpu scratch race replica */
static struct dst_entry *fr_route_dst_new(void)
{
struct dst_entry *d = dst_alloc(&fr_dst_ops, fr_ndev,
DST_OBSOLETE_NONE, DST_NOCOUNT);
if (d)
atomic_inc(&fr_dst_created);
return d;
}
/* replica of bpf_skb_set_xfrm_info() (xfrm_interface_bpf.c:64-94) */
static int fr_set_xfrm_info(struct sk_buff *skb, u32 if_id, int link)
{
struct metadata_dst *md_dst;
struct xfrm_md_info *info;
if (unlikely(skb_metadata_dst(skb)))
return -EINVAL;
md_dst = this_cpu_ptr(fr_scratch);
info = &md_dst->u.xfrm_info;
info->if_id = if_id;
info->link = link;
skb_dst_force(skb);
info->dst_orig = skb_dst(skb); /* unconditional overwrite */
dst_hold((struct dst_entry *)md_dst);
skb_dst_set(skb, (struct dst_entry *)md_dst);
return 0;
}
/* replica of xfrmi_xmit2() collect_md block (xfrm_interface_core.c:434-455) */
static void fr_xmit2_swap(struct sk_buff *skb)
{
struct xfrm_md_info *md_info = skb_xfrm_md_info(skb);
struct dst_entry *dst;
if (!md_info)
return;
if (md_info->dst_orig) { /* read-take-clear, non-atomic */
struct dst_entry *tmp_dst = skb_dst(skb);
dst = md_info->dst_orig;
skb_dst_set(skb, dst);
md_info->dst_orig = NULL;
dst_release(tmp_dst); /* drop this skb's scratch ref */
}
}
static int fr_casec_setup(void)
{
fr_dst_cache = kmem_cache_create("frdst", sizeof(struct dst_entry),
0, 0, NULL);
if (!fr_dst_cache)
return -ENOMEM;
fr_dst_ops.kmem_cachep = fr_dst_cache;
fr_dst_ops.destroy = fr_dst_destroy;
fr_scratch = metadata_dst_alloc_percpu(0, METADATA_XFRM, GFP_KERNEL);
if (!fr_scratch)
return -ENOMEM;
return 0;
}
static void fr_casec_teardown(void)
{
if (fr_scratch) {
metadata_dst_free_percpu(fr_scratch);
fr_scratch = NULL;
}
if (fr_dst_cache) {
kmem_cache_destroy(fr_dst_cache);
fr_dst_cache = NULL;
}
}
static int fr_do_casec_ctl(void)
{
struct dst_entry *r1, *r2, *rx;
struct sk_buff *s1, *s2, *s3;
int before, leaks;
if (fr_casec_setup())
return -ENOMEM;
/* --- baseline: single-packet protocol closes cleanly --- */
r1 = fr_route_dst_new();
s1 = alloc_skb(64, GFP_KERNEL);
skb_dst_set(s1, r1); /* skb owns r1's single ref */
preempt_disable(); /* keep this_cpu_ptr stable */
fr_set_xfrm_info(s1, 1, 0);
fr_xmit2_swap(s1);
preempt_enable();
kfree_skb(s1); /* drops r1 -> freed via RCU */
synchronize_rcu();
pr_info(FRTAG " casec baseline: created=%d destroyed=%d (expect 1/1)\n",
atomic_read(&fr_dst_created), atomic_read(&fr_dst_destroyed));
/* --- leak variant: unconditional overwrite of pending dst_orig --- */
before = atomic_read(&fr_dst_destroyed);
r1 = fr_route_dst_new();
r2 = fr_route_dst_new();
s1 = alloc_skb(64, GFP_KERNEL);
s2 = alloc_skb(64, GFP_KERNEL);
skb_dst_set(s1, r1);
skb_dst_set(s2, r2);
preempt_disable();
fr_set_xfrm_info(s1, 1, 0); /* scratch.dst_orig = r1 */
fr_set_xfrm_info(s2, 2, 0); /* OVERWRITES: r1 leaked */
fr_xmit2_swap(s1); /* gets r2 (cross-packet) */
fr_xmit2_swap(s2); /* dst_orig==NULL -> skip */
preempt_enable();
kfree_skb(s1);
kfree_skb(s2);
synchronize_rcu();
leaks = atomic_read(&fr_dst_created) - atomic_read(&fr_dst_destroyed);
pr_info(FRTAG " CASEC-LEAK: leaked=%d (r1 refs_held=%d, expected 0 if not leaked; baseline destroyed+%d of 2)\n",
leaks, rcuref_read(&r1->__rcuref),
atomic_read(&fr_dst_destroyed) - before);
dst_release(r1); /* reclaim the leaked ref */
/* --- double-consume: interleaved read-take-clear (the exact
* instruction order two concurrent xfrmi_xmit2 executions produce;
* natural reachability is shown by casec_nat) --- */
rx = fr_route_dst_new(); /* refcnt = 1 */
s1 = alloc_skb(64, GFP_KERNEL);
s2 = alloc_skb(64, GFP_KERNEL);
s3 = alloc_skb(64, GFP_KERNEL);
skb_dst_set(s3, rx); /* the "original skb" hands rx */
preempt_disable();
fr_set_xfrm_info(s3, 3, 0); /* scratch.dst_orig = rx */
{
struct xfrm_md_info *mi = this_cpu_ptr(&fr_scratch->u.xfrm_info);
struct dst_entry *t1, *t2;
t1 = mi->dst_orig; /* skb1 xmit2: read */
t2 = mi->dst_orig; /* skb2 xmit2: read */
skb_dst_set(s1, t1); /* skb1 takes */
skb_dst_set(s2, t2); /* skb2 takes - 1 ref, 2 owners */
mi->dst_orig = NULL;
}
preempt_enable();
kfree_skb(s3); /* drops scratch md ref */
pr_info(FRTAG " casec dcons: rx refs_held=%d (2 claims on 1 real ref)\n",
rcuref_read(&rx->__rcuref));
kfree_skb(s1); /* first owner releases */
pr_info(FRTAG " casec dcons: after skb1 free rx dead=%d\n",
rcuref_is_dead(&rx->__rcuref));
kfree_skb(s2); /* second owner: imbalanced put */
synchronize_rcu(); /* dst_destroy_rcu frees rx */
pr_info(FRTAG " CASEC-DCONS: touching rx after grace period (UAF read expected below)\n");
pr_info(FRTAG " CASEC-DCONS: rx->obsolete=%d created=%d destroyed=%d\n",
rx->obsolete, atomic_read(&fr_dst_created),
atomic_read(&fr_dst_destroyed));
fr_casec_teardown();
return 0;
}
/* natural churn: two threads on the SAME cpu (shared percpu scratch) */
static atomic_t fr_cc_cycles, fr_cc_confused, fr_cc_skipped;
static int fr_cc_fn(void *unused)
{
int i;
for (i = 0; i < iters; i++) {
struct dst_entry *r = fr_route_dst_new();
struct sk_buff *skb = alloc_skb(64, GFP_KERNEL);
int us;
if (r && skb) {
skb_dst_set(skb, r);
fr_set_xfrm_info(skb, i & 1, 0);
us = get_random_u32_below(jitter_us + 1);
if (us)
udelay(us); /* widen set->xmit window */
if (skb_dst(skb) == (struct dst_entry *)
this_cpu_ptr(fr_scratch)) {
struct dst_entry *want = r;
fr_xmit2_swap(skb);
if (skb_dst(skb) != want) {
if (skb_dst(skb) == (struct dst_entry *)
this_cpu_ptr(fr_scratch))
atomic_inc(&fr_cc_skipped);
else
atomic_inc(&fr_cc_confused);
}
}
kfree_skb(skb);
} else {
if (skb)
kfree_skb(skb);
if (r)
dst_release(r);
}
atomic_inc(&fr_cc_cycles);
cond_resched();
}
complete(&fr_cc_done);
return 0;
}
static void fr_scratch_drain(void)
{
int cpu;
/* replicate metadata_dst_free_percpu()'s dst_orig release loop so
* accounting is settled before the final destroy-count read */
for_each_possible_cpu(cpu) {
struct xfrm_md_info *mi =
per_cpu_ptr(&fr_scratch->u.xfrm_info, cpu);
if (mi->dst_orig) {
dst_release(mi->dst_orig);
mi->dst_orig = NULL;
}
}
}
static int fr_do_casec_nat(void)
{
struct task_struct *t1, *t2;
int leaked;
if (fr_casec_setup())
return -ENOMEM;
init_completion(&fr_cc_done);
t1 = kthread_create(fr_cc_fn, NULL, "fr_cc0");
t2 = kthread_create(fr_cc_fn, NULL, "fr_cc1");
if (IS_ERR(t1) || IS_ERR(t2))
return -ENOMEM;
kthread_bind(t1, 1); /* same CPU -> same scratch */
kthread_bind(t2, 1);
wake_up_process(t1);
wake_up_process(t2);
/* let both run their full bounded loops, then settle accounting */
wait_for_completion(&fr_cc_done);
wait_for_completion(&fr_cc_done);
fr_scratch_drain();
synchronize_rcu();
leaked = atomic_read(&fr_dst_created) - atomic_read(&fr_dst_destroyed);
pr_info(FRTAG " CASEC-NAT: cycles=%d created=%d destroyed=%d leaked=%d confused(other R)=%d swap_skipped=%d\n",
atomic_read(&fr_cc_cycles), atomic_read(&fr_dst_created),
atomic_read(&fr_dst_destroyed), leaked,
atomic_read(&fr_cc_confused), atomic_read(&fr_cc_skipped));
fr_casec_teardown();
return 0;
}
/* ------------------------------------------------------------------ */
static int __init flowrace_init(void)
{
int ret = 0;
fr_ndev = dev_get_by_name(&init_net, "lo");
if (!fr_ndev) {
pr_err(FRTAG " no lo netdev\n");
return -ENODEV;
}
pr_info(FRTAG " action=%s iters=%d readers=%d jitter_us=%d\n",
action, iters, nreaders, jitter_us);
if (!strcmp(action, "sanity"))
ret = fr_do_sanity();
else if (!strcmp(action, "casea_ctl"))
ret = fr_do_casea_ctl();
else if (!strcmp(action, "casea_nat"))
ret = fr_do_casea_nat();
else if (!strcmp(action, "casea_nat2"))
ret = fr_do_casea_nat2();
else if (!strcmp(action, "caseb"))
ret = fr_do_caseb();
else if (!strcmp(action, "casec_ctl"))
ret = fr_do_casec_ctl();
else if (!strcmp(action, "casec_nat"))
ret = fr_do_casec_nat();
else
ret = -EINVAL;
dev_put(fr_ndev);
pr_info(FRTAG "-DONE action=%s ret=%d cb_calls=%d\n", action, ret,
atomic_read(&fr_cb_calls));
return ret;
}
static void __exit flowrace_exit(void)
{
}
module_init(flowrace_init);
module_exit(flowrace_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("net/core flow_indr / xfrm scratch race verification");