io_uring/zcrx: copy statistics lose updates when one ifq is shared by multiple rings (Data race)
From: 徐晗翔
Date: Sun Nov 22 2026 - 02:39:55 EST
An io_uring zero-copy RX ifq can be exported by one ring and imported by
another. Both rings then perform receive-zc on their own TCP sockets,
but they share the same struct io_zcrx_ifq and therefore the same
ifq->notif_stats. The statistics update is not atomic:
static void zcrx_stat_add(__u64 *p, s64 v)
{
WRITE_ONCE(*p, READ_ONCE(*p) + v);
}
io_uring/zcrx.c:1337-1340. It is called for copy_count and copy_bytes
from io_zcrx_copy_frag() at zcrx.c:1770-1771. There is no ifq-level
lock in this path. lock_sock() in io_zcrx_tcp_recvmsg() is per TCP
socket, the ring's uring_lock is per ring, alloc_lock only protects
fallback niov allocation and is released before the copy, and
ctx_lock/notification is taken after the statistics update and is not
involved when event_desc.type_mask == 0. Two CPUs can therefore read
the same old value, add their own increments, and write back the same
new value: one update is lost.
The effect is limited to the UAPI cumulative copy statistics
(struct zcrx_stats at include/uapi/linux/io_uring/zcrx.h:83-86). The
values are user-mappable diagnostic data; the race does not control
buffer lifetime, DMA, reference counts, or packet contents.
The detailed description:
* Test environment / affected version
Linux 7.3.0 (gcc 13.3.0, GNU ld 2.42) #6 SMP PREEMPT_DYNAMIC,
vanilla mainline 7.3, x86_64 QEMU/KVM guest, 8 vCPUs, 4 GiB,
CONFIG_KASAN=y. The tested io_uring/zcrx.c is byte-identical to
v7.3: SHA-256
6e4e9828c84179b7a1877f0f843cc230f865d42665cbd0d43441493c67c70980.
The kernel was not modified; no delay or other amplification was
inserted. bzImage SHA-256:
cdc1122bc0bdd2c65f92650bef919ff7b1f15d42bf64e3b0b079352fccf7b104.
Relevant config includes CONFIG_IO_URING=y, CONFIG_IO_URING_ZCRX=y
and CONFIG_KASAN=y.
* Mechanism
Shared object and two writers:
- Ring A registers a ZCRX_REG_NODEV ifq with a user-memory area and
a stats-only event descriptor: event_desc.flags =
ZCRX_EVENT_DESC_FLAG_STATS and event_desc.type_mask = 0. The
stats pointer is validated and stored at zcrx.c:935-960 (pointer
at 956, memset at 957). Because type_mask is zero,
zcrx_set_ring_ctx() is not called (zcrx.c:1092-1093) and
master_ctx is NULL; this is valid and statistics still update.
- A exports the ifq with ZCRX_CTRL_EXPORT. zcrx_export() checks
that the control fields are zero and increments refs and
user_refs at zcrx.c:805-806, attaches the original ifq to the
[zcrx] file at 808-809, and installs an fd.
- Ring B imports the same ifq with ZCRX_REG_IMPORT. import_zcrx()
requires DEFER_TASKRUN plus CQE32 or CQE_MIXED, rejects
nonzero if_rxq/rq_entries/area_ptr/region_ptr/event_desc, takes
the same ifq at zcrx.c:861, increments refs/user_refs at 862-863,
and stores it in B's xarray at 880.
- A and B each submit a multishot IORING_OP_RECV_ZC on a different
TCP socket. A's and B's recv paths can execute
io_zcrx_copy_frag() concurrently.
The missing serialization:
- zcrx_stat_add() is a plain load-add-store, not an atomic RMW.
- lock_sock(sk) at zcrx.c:1927 only serializes operations on one
socket; A and B use different sockets.
- uring_lock is per ring and does not cross rings.
- alloc_lock at zcrx.c:1663 only covers io_alloc_fallback_niov();
it is not held during the copy or the statistics update.
- ctx_lock at zcrx.c:1351 protects notification state and
master_ctx, is taken after the statistics update, and is not
reached at all for type_mask == 0.
- no pp_lock or rq.lock is held by the copy path.
Minimal interleaving for one field:
CPU 0: read copy_bytes == V
CPU 1: read copy_bytes == V
CPU 0: write V + len0
CPU 1: write V + len1 (len0 is lost)
copy_count has the same pattern with v = 1. The final values can be
lower than the number/bytes of data CQEs actually delivered, and the
two fields can be mutually inconsistent.
The paged-fragment requirement is important: the linear skb head is
handled by io_zcrx_copy_chunk() (zcrx.c:1710-1758) and does not
update stats. A non-net_iov paged fragment enters
io_zcrx_copy_frag() through io_zcrx_recv_frag() at zcrx.c:1785-1786,
which is where zcrx_stat_add() is called. A pure linear send may
therefore produce CQEs without exercising the race.
* Trigger model
Requirements:
1. Two rings, each created with DEFER_TASKRUN, SINGLE_ISSUER and
CQE32 or CQE_MIXED (the io_uring_setup requirement for
DEFER_TASKRUN is in io_uring/io_uring.c:2832-2833).
2. Ring A has a stat-enabled NODEV ifq with a user-memory area and
a valid stats_offset; event_desc.type_mask may be zero.
3. A exports the ifq and B imports it, so both use the same ifq and
the same notif_stats address.
4. A and B receive on different TCP sockets (different lock_sock
instances).
5. Paged non-net_iov skb fragments enter io_zcrx_copy_frag(); a
sendfile()/splice/MSG_ZEROCOPY style sender on loopback is
sufficient in the reproducer.
6. The two updates overlap in the load-add-store window.
No special hardware is needed for the NODEV path; a loopback TCP pair
is enough. The existing reproducer uses two independent loopback
connections and two sender threads.
* Reproducer (attached: zcrx_verify.c) and results on 7.3
zcrx_verify.c is a self-contained raw-syscall program; no liburing is
required. It creates two rings A/B, registers the NODEV ifq on A,
exports it, imports it on B, drives one multishot IORING_OP_RECV_ZC
per ring, returns each data CQE's rcqe->off as an RQE, and calls
IORING_REGISTER_ZCRX_CTRL with ZCRX_CTRL_FLUSH_RQ on the owning ring.
FLUSH_RQ is required on the NODEV path: there is no page_pool
automatic refill, so without it the area is exhausted and the run
stops. FLUSH_RQ must be issued by the thread that owns the ring
(otherwise the kernel returns -EEXIST).
Build:
gcc -static -O2 -g -pthread -Wall -Wextra \
-Ilinux-7.3/include/uapi -Ilinux-7.3/include \
-o zcrx_verify zcrx_verify.c
Run: boot the 7.3 KASAN bzImage under QEMU/KVM with a custom
initramfs whose /init runs /zcrx_verify, with 8 vCPUs and 4 GiB.
The program takes the run time in seconds as its argument. The
dual-ring log was captured with a 0-error clean run; SINGLE=1 runs
the same binary with only ring A as a control.
Results (unmodified v7.3, no test delay):
single ring, zcrx-single3.log:
data_cqes = 223000, data_bytes = 859635712
copy_count = 223000, copy_bytes = 859635712
delta_count = 0, delta_bytes = 0
dual ring, zcrx-final-clean.log (0 error CQEs):
data_cqes = 76888, data_bytes = 296353792
copy_count = 76863, copy_bytes = 296243359
delta_count = -25, delta_bytes = -110433
additional dual-ring trials:
-8245 bytes / -3 count, -24470 bytes / -6 count,
-77824 bytes / -17 count
The single-ring control is exact at 223,000 CQEs / 859 MB, so the
accounting convention is consistent and there is no systemic
linear-head or CQ-loss bias in this test. The dual-ring runs are
consistently negative; the clean run has zero error CQEs and still
loses 110,433 bytes and 25 count increments. copy_count counts
per-fragment updates while the data CQE count is per completed
buffer, so the count delta is not expected to be numerically equal
to the CQE delta; copy_bytes is the primary signal. No KASAN
report, WARN or Oops was observed.
* Suggested direction
Serialize the two updates per ifq, e.g. add a spinlock_t stats_lock
to struct io_zcrx_ifq, initialize it in io_zcrx_ifq_alloc(), and
hold it around both zcrx_stat_add() calls in io_zcrx_copy_frag(),
releasing it before zcrx_send_notif(). A lockless alternative is to
keep private atomic64 counters and publish a consistent snapshot, or
to document the fields as best-effort. READ_ONCE()/WRITE_ONCE() alone
do not make the read-modify-write atomic and do not fix the lost
update.
Please keep me in Cc; I am not subscribed to the lists.
Regards,
Hanxiang Xu
xuhx@xxxxxxxxxxx
// Test-only PoC for io_uring ZCRX shared-ifq copy statistics lost update.
// Runs in the verification VM as root. Uses raw syscalls (no liburing).
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <inttypes.h>
#include <linux/io_uring.h>
#include <linux/io_uring/zcrx.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <pthread.h>
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/sendfile.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
#include <net/if.h>
#ifndef __NR_io_uring_setup
#define __NR_io_uring_setup 425
#endif
#ifndef __NR_io_uring_enter
#define __NR_io_uring_enter 426
#endif
#ifndef __NR_io_uring_register
#define __NR_io_uring_register 427
#endif
#define PAGE 4096UL
#define AREA_PAGES 16384
#define AREA_SIZE (PAGE * AREA_PAGES)
#define REGION_SIZE 262144
#define RQ_ENTRIES 4096
#define STATS_OFF 131072
#define CQE_SIZE 32
struct ring {
int fd;
unsigned *sq_head, *sq_tail, *sq_mask, *sq_array;
struct io_uring_sqe *sqes;
unsigned *cq_head, *cq_tail, *cq_mask;
char *cqes;
unsigned entries;
};
static int io_uring_setup(unsigned e, struct io_uring_params *p) { return syscall(__NR_io_uring_setup, e, p); }
static int io_uring_enter(int fd, unsigned to_submit, unsigned min_complete, unsigned flags) {
return syscall(__NR_io_uring_enter, fd, to_submit, min_complete, flags, NULL, 0);
}
static int io_uring_register(int fd, unsigned op, void *arg, unsigned nr) {
return syscall(__NR_io_uring_register, fd, op, arg, nr);
}
static int setup_ring(struct ring *r, unsigned entries, unsigned flags)
{
struct io_uring_params p;
memset(&p, 0, sizeof(p));
p.flags = flags;
r->fd = io_uring_setup(entries, &p);
if (r->fd < 0) { perror("io_uring_setup"); return -1; }
r->entries = p.sq_entries;
size_t sq_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned);
size_t cq_sz = p.cq_off.cqes + p.cq_entries * CQE_SIZE;
size_t map_sz = sq_sz > cq_sz ? sq_sz : cq_sz;
char *base = mmap(NULL, map_sz, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, r->fd, IORING_OFF_SQ_RING);
if (base == MAP_FAILED) { perror("mmap sq"); return -1; }
char *cqbase;
if (p.features & IORING_FEAT_SINGLE_MMAP) {
cqbase = base;
} else {
cqbase = mmap(NULL, cq_sz, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, r->fd, IORING_OFF_CQ_RING);
if (cqbase == MAP_FAILED) { perror("mmap cq"); return -1; }
}
r->sq_head = (unsigned *)(base + p.sq_off.head);
r->sq_tail = (unsigned *)(base + p.sq_off.tail);
r->sq_mask = (unsigned *)(base + p.sq_off.ring_mask);
r->sq_array = (unsigned *)(base + p.sq_off.array);
r->sqes = mmap(NULL, p.sq_entries * sizeof(struct io_uring_sqe), PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_POPULATE, r->fd, IORING_OFF_SQES);
if (r->sqes == MAP_FAILED) { perror("mmap sqes"); return -1; }
r->cq_head = (unsigned *)(cqbase + p.cq_off.head);
r->cq_tail = (unsigned *)(cqbase + p.cq_off.tail);
r->cq_mask = (unsigned *)(cqbase + p.cq_off.ring_mask);
r->cqes = cqbase + p.cq_off.cqes;
printf("ring fd=%d entries=%u cq=%u feat=0x%x single=%d\n", r->fd, p.sq_entries, p.cq_entries,
p.features, !!(p.features & IORING_FEAT_SINGLE_MMAP));
return 0;
}
static void submit_recvzc(struct ring *r, int sock, unsigned ifq_idx, uint64_t ud)
{
unsigned tail = __atomic_load_n(r->sq_tail, __ATOMIC_RELAXED);
unsigned idx = tail & *r->sq_mask;
struct io_uring_sqe *sqe = &r->sqes[idx];
memset(sqe, 0, sizeof(*sqe));
sqe->opcode = IORING_OP_RECV_ZC;
sqe->fd = sock;
sqe->len = 0;
sqe->ioprio = IORING_RECV_MULTISHOT;
sqe->zcrx_ifq_idx = ifq_idx;
sqe->user_data = ud;
r->sq_array[idx] = idx;
__atomic_store_n(r->sq_tail, tail + 1, __ATOMIC_RELEASE);
int ret = io_uring_enter(r->fd, 1, 0, 0);
if (ret < 0) fprintf(stderr, "enter submit: %s\n", strerror(errno));
}
static int make_loop_pair(int *rx, int *tx)
{
int lfd = socket(AF_INET, SOCK_STREAM, 0);
if (lfd < 0) { perror("socket l"); return -1; }
int one = 1;
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
struct sockaddr_in a; memset(&a, 0, sizeof(a));
a.sin_family = AF_INET; a.sin_addr.s_addr = htonl(INADDR_LOOPBACK); a.sin_port = 0;
if (bind(lfd, (void *)&a, sizeof(a)) < 0) { perror("bind"); return -1; }
if (listen(lfd, 8) < 0) { perror("listen"); return -1; }
socklen_t alen = sizeof(a);
if (getsockname(lfd, (void *)&a, &alen) < 0) { perror("getsockname"); return -1; }
int c = socket(AF_INET, SOCK_STREAM, 0);
if (c < 0) { perror("socket c"); return -1; }
if (connect(c, (void *)&a, sizeof(a)) < 0) { perror("connect"); return -1; }
int s = accept(lfd, NULL, NULL);
if (s < 0) { perror("accept"); return -1; }
close(lfd);
setsockopt(c, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one));
setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one));
*rx = c; *tx = s;
return 0;
}
struct zcrx {
struct ring ring;
unsigned ifq_id;
unsigned rq_entries;
uint64_t area_token;
char *region;
uint32_t *rq_tail;
struct io_uring_zcrx_rqe *rqes;
uint64_t *stats;
pthread_mutex_t *rq_lock;
pthread_mutex_t rq_lock_storage;
int ctrl_fd;
volatile int *stop;
uint64_t cqes_data;
uint64_t bytes_data;
int errors;
};
static void return_cqe(struct zcrx *z, struct io_uring_cqe *cqe)
{
struct io_uring_zcrx_cqe *rcqe = (struct io_uring_zcrx_cqe *)(cqe + 1);
uint64_t off = rcqe->off;
int res = cqe->res;
if (res <= 0) return;
pthread_mutex_lock(z->rq_lock);
uint32_t tail = __atomic_load_n(z->rq_tail, __ATOMIC_RELAXED);
struct io_uring_zcrx_rqe *rqe = &z->rqes[tail & (z->rq_entries - 1)];
rqe->off = (off & ~IORING_ZCRX_AREA_MASK) | z->area_token;
rqe->len = (uint32_t)res;
rqe->__pad = 0;
__atomic_store_n(z->rq_tail, tail + 1, __ATOMIC_RELEASE);
struct zcrx_ctrl ctrl; memset(&ctrl, 0, sizeof(ctrl));
ctrl.zcrx_id = z->ifq_id;
ctrl.op = ZCRX_CTRL_FLUSH_RQ;
int ret = io_uring_register(z->ctrl_fd, IORING_REGISTER_ZCRX_CTRL, &ctrl, 0);
if (ret < 0) {
if (z->errors++ < 5) fprintf(stderr, "flush err: %s\n", strerror(errno));
}
pthread_mutex_unlock(z->rq_lock);
}
static void *recv_loop(void *arg)
{
struct zcrx *z = arg;
unsigned head = __atomic_load_n(z->ring.cq_head, __ATOMIC_RELAXED);
while (!*z->stop) {
int ret = io_uring_enter(z->ring.fd, 0, 0, IORING_ENTER_GETEVENTS);
if (ret < 0) {
if (errno == EINTR || errno == EAGAIN) { usleep(50); continue; }
fprintf(stderr, "enter wait: %s\n", strerror(errno));
usleep(50);
continue;
}
unsigned tail = __atomic_load_n(z->ring.cq_tail, __ATOMIC_ACQUIRE);
while (head != tail) {
struct io_uring_cqe *cqe = (struct io_uring_cqe *)(z->ring.cqes + (head & *z->ring.cq_mask) * CQE_SIZE);
if (cqe->res > 0) {
z->cqes_data++;
z->bytes_data += (uint64_t)cqe->res;
return_cqe(z, cqe);
} else if (cqe->res < 0 && z->errors++ < 5) {
fprintf(stderr, "cqe res=%d flags=0x%x\n", cqe->res, cqe->flags);
}
head++;
}
__atomic_store_n(z->ring.cq_head, head, __ATOMIC_RELEASE);
usleep(50);
}
return NULL;
}
struct sender_arg { int fd; int file_fd; volatile int *stop; uint64_t bytes; };
static void *send_loop(void *arg)
{
struct sender_arg *s = arg;
char buf[65536];
off_t off = 0;
while (!*s->stop) {
ssize_t n = sendfile(s->fd, s->file_fd, &off, sizeof(buf));
if (n > 0) { s->bytes += n; continue; }
if (n < 0 && (errno == EAGAIN || errno == EINTR)) continue;
off = 0;
if (n == 0) continue;
break;
}
return NULL;
}
static void bring_lo_up(void)
{
int fd = socket(AF_INET, SOCK_DGRAM, 0);
struct ifreq ifr; memset(&ifr, 0, sizeof(ifr));
strcpy(ifr.ifr_name, "lo");
if (ioctl(fd, SIOCGIFFLAGS, &ifr) == 0) {
ifr.ifr_flags |= IFF_UP | IFF_RUNNING;
ioctl(fd, SIOCSIFFLAGS, &ifr);
}
close(fd);
}
static int make_data_file(void)
{
int fd = open("/tmp/zcrx-data", O_CREAT|O_RDWR|O_TRUNC, 0600);
if (fd < 0) { perror("open data"); return -1; }
char *buf = malloc(1 << 20);
if (!buf) return -1;
for (int i = 0; i < (1 << 20); i++) buf[i] = (char)(i * 7 + 1);
if (write(fd, buf, 1 << 20) != (1 << 20)) { perror("write data"); return -1; }
free(buf);
lseek(fd, 0, SEEK_SET);
return fd;
}
struct b_arg {
struct zcrx *z;
int export_fd;
int rx;
unsigned flags;
volatile int *ready;
};
static void *b_thread(void *arg)
{
struct b_arg *a = arg;
struct zcrx *z = a->z;
if (setup_ring(&z->ring, 4096, a->flags) < 0) { *a->ready = -1; return NULL; }
struct io_uring_zcrx_ifq_reg ireg; memset(&ireg, 0, sizeof(ireg));
ireg.if_idx = a->export_fd; ireg.flags = ZCRX_REG_IMPORT;
int r = io_uring_register(z->ring.fd, IORING_REGISTER_ZCRX_IFQ, &ireg, 1);
if (r < 0) { fprintf(stderr, "import: %s\n", strerror(errno)); *a->ready = -1; return NULL; }
z->ifq_id = ireg.zcrx_id;
z->ctrl_fd = z->ring.fd;
submit_recvzc(&z->ring, a->rx, z->ifq_id, 0xBBBB);
*a->ready = 1;
recv_loop(z);
return NULL;
}
struct stop_arg { volatile int *stop; int fds[4]; int secs; };
static void *timer_thread(void *arg)
{
struct stop_arg *s = arg;
sleep(s->secs);
*s->stop = 1;
for (int i = 0; i < 4; i++) if (s->fds[i] >= 0) shutdown(s->fds[i], SHUT_RDWR);
return NULL;
}
int main(int argc, char **argv)
{
int secs = argc > 1 ? atoi(argv[1]) : 8;
int single = getenv("SINGLE") != NULL;
signal(SIGPIPE, SIG_IGN);
bring_lo_up();
/* Ring A: owns the ifq, stats-only, NODEV, user area. */
struct ring ra;
unsigned flags = IORING_SETUP_CQE32 | IORING_SETUP_SINGLE_ISSUER | IORING_SETUP_DEFER_TASKRUN;
if (setup_ring(&ra, 4096, flags) < 0) return 1;
char *region = mmap(NULL, REGION_SIZE, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_POPULATE, -1, 0);
if (region == MAP_FAILED) { perror("mmap region"); return 1; }
memset(region, 0, REGION_SIZE);
uint64_t *stats = (uint64_t *)(region + STATS_OFF);
char *area_mem;
if (posix_memalign((void **)&area_mem, PAGE, AREA_SIZE)) { perror("posix_memalign"); return 1; }
memset(area_mem, 0x5a, AREA_SIZE);
struct io_uring_region_desc rd; memset(&rd, 0, sizeof(rd));
rd.user_addr = (uint64_t)region; rd.size = REGION_SIZE; rd.flags = IORING_MEM_REGION_TYPE_USER;
struct io_uring_zcrx_area_reg area; memset(&area, 0, sizeof(area));
area.addr = (uint64_t)area_mem; area.len = AREA_SIZE;
struct zcrx_event_desc ev; memset(&ev, 0, sizeof(ev));
ev.type_mask = 0; ev.flags = ZCRX_EVENT_DESC_FLAG_STATS; ev.stats_offset = STATS_OFF;
struct io_uring_zcrx_ifq_reg reg; memset(®, 0, sizeof(reg));
reg.rq_entries = RQ_ENTRIES; reg.flags = ZCRX_REG_NODEV;
reg.area_ptr = (uint64_t)&area; reg.region_ptr = (uint64_t)&rd; reg.event_desc = (uint64_t)&ev;
int ret = io_uring_register(ra.fd, IORING_REGISTER_ZCRX_IFQ, ®, 1);
if (ret < 0) { fprintf(stderr, "register ifq: %s\n", strerror(errno)); return 1; }
printf("IFQ registered id=%u rx_buf_len=%u rqes_off=%u area_token=%#llx\n",
reg.zcrx_id, reg.rx_buf_len, reg.offsets.rqes,
(unsigned long long)area.rq_area_token);
struct zcrx_ctrl ctrl; memset(&ctrl, 0, sizeof(ctrl));
ctrl.zcrx_id = reg.zcrx_id; ctrl.op = ZCRX_CTRL_EXPORT;
ret = io_uring_register(ra.fd, IORING_REGISTER_ZCRX_CTRL, &ctrl, 0);
if (ret < 0) { fprintf(stderr, "export: %s\n", strerror(errno)); return 1; }
int export_fd = ctrl.zc_export.zcrx_fd;
printf("export fd=%d\n", export_fd);
/* Two TCP pairs: A and B read from different sockets. */
int rx_a, tx_a, rx_b, tx_b;
if (make_loop_pair(&rx_a, &tx_a) || make_loop_pair(&rx_b, &tx_b)) return 1;
int file_fd = make_data_file();
if (file_fd < 0) return 1;
struct zcrx za; memset(&za, 0, sizeof(za));
za.ring = ra; za.ifq_id = reg.zcrx_id; za.rq_entries = RQ_ENTRIES;
za.area_token = area.rq_area_token; za.region = region;
za.rqes = (struct io_uring_zcrx_rqe *)(region + reg.offsets.rqes);
za.rq_tail = (uint32_t *)(region + reg.offsets.tail);
za.stats = stats;
volatile int stop = 0;
volatile int b_ready = 0;
za.stop = &stop;
za.ctrl_fd = ra.fd;
pthread_mutex_init(&za.rq_lock_storage, NULL);
za.rq_lock = &za.rq_lock_storage;
/* Ring B imports the same ifq. It must be created in the B thread for SINGLE_ISSUER. */
struct zcrx zb; memset(&zb, 0, sizeof(zb));
zb.ifq_id = reg.zcrx_id; zb.rq_entries = RQ_ENTRIES; zb.area_token = area.rq_area_token;
zb.region = region; zb.rqes = za.rqes; zb.rq_tail = za.rq_tail; zb.stats = stats;
zb.stop = &stop;
zb.rq_lock = &za.rq_lock_storage;
zb.ctrl_fd = ra.fd;
struct sender_arg sa = { .fd = tx_a, .file_fd = file_fd, .stop = &stop };
struct sender_arg sb = { .fd = tx_b, .file_fd = file_fd, .stop = &stop };
pthread_t tb = 0, tsa, tsb = 0, tt;
pthread_create(&tsa, NULL, send_loop, &sa);
if (!single) pthread_create(&tsb, NULL, send_loop, &sb);
struct b_arg ba = { .z = &zb, .export_fd = export_fd, .rx = rx_b, .flags = flags, .ready = &b_ready };
if (!single) pthread_create(&tb, NULL, b_thread, &ba);
else __atomic_store_n(&b_ready, 1, __ATOMIC_RELEASE);
while (!__atomic_load_n(&b_ready, __ATOMIC_ACQUIRE) && !stop) usleep(1000);
submit_recvzc(&ra, rx_a, reg.zcrx_id, 0xAAAA);
struct stop_arg st = { .stop = &stop, .fds = { rx_a, tx_a, single ? -1 : rx_b, single ? -1 : tx_b }, .secs = secs };
pthread_create(&tt, NULL, timer_thread, &st);
recv_loop(&za);
pthread_join(tt, NULL);
if (!single) pthread_join(tb, NULL);
pthread_join(tsa, NULL);
if (!single) pthread_join(tsb, NULL);
printf("RESULT data_cqes=%llu data_bytes=%llu copy_count=%llu copy_bytes=%llu\n",
(unsigned long long)(za.cqes_data + zb.cqes_data),
(unsigned long long)(za.bytes_data + zb.bytes_data),
(unsigned long long)stats[0], (unsigned long long)stats[1]);
printf("RESULT A cqes=%llu bytes=%llu B cqes=%llu bytes=%llu errors=%d/%d\n",
(unsigned long long)za.cqes_data, (unsigned long long)za.bytes_data,
(unsigned long long)zb.cqes_data, (unsigned long long)zb.bytes_data,
za.errors, zb.errors);
printf("RESULT delta_bytes=%lld delta_count_vs_cqes=%lld\n",
(long long)((int64_t)stats[1] - (int64_t)(za.bytes_data + zb.bytes_data)),
(long long)((int64_t)stats[0] - (int64_t)(za.cqes_data + zb.cqes_data)));
return 0;
}