Re: [PATCH RFC 00/18] scsi: target: Add target_core_uring, an io_uring based user backstore

From: Qiliang Yuan

Date: Wed Nov 18 2026 - 20:03:12 EST


On Tue, Nov 17, 2026 at 07:40:18PM -0600, Ming Lei wrote:
> Can you share your setup? It is supposed that you run fio from VM,
> otherwise you can just compare ublk with scsi backed with tcmu.
>
> But if you run fio from VM, I believe you can setup virtio-scsi backed by
> ublk block device directly, which should cover your requirement, or maybe
> I miss something?

Everything runs in one 32 CPU guest. For tcm_loop and iSCSI (16
sessions over 127.0.0.1), fio runs in that guest on the initiator
disks. For vhost-scsi, fio runs in a nested 16 vCPU KVM guest on the
vhost-scsi disk, with the target and the backstore server in the outer
guest. In the ublk runs, kublk serves the file as /dev/ublkbN and
target_core_iblock exports that through the same fabric.

You are not missing anything. This backstore has block semantics: the
kernel emulates the SCSI commands, PR and ALUA included, exactly as for
iblock, so ublk behind target_core_iblock offers the same function on
every fabric, and for a VM a virtio-scsi or vhost-scsi disk on top of
the ublk device covers it as well. What the series adds over that is
only performance, and outside tcm_loop the numbers I sent do not show a
consistent gain.

[...]

> My question is actually why there is regression wrt. 4k randread:
>
> workload TCM/USER (MB/s) uring (MB/s) change
> 4k randread, 1 job 426 (409-463) 369 (366-384) -13%
>
> Maybe you can focus on single job perf only, which is easy to compare and
> investigate.

It comes from the scheduler moving fio between CPUs. The pwrite() I
pointed to in my last mail is not it: doing that copy inside
io_uring_enter() instead makes no difference.

tcm_loop queues the LIO submission and completion work on the CPU that
submitted the command, so with one job, fio's CPU always has bound
kworkers and ksoftirqd runnable next to fio. The load balancer cannot
move those and moves fio instead. This backstore gives it more chances:
the completion work on fio's CPU handles 14 commands per run here
against 24 with TCM/USER, so per 1000 I/Os the completion kworker and
ksoftirqd run about 57 times against 35, fio is preempted as often, and
it is moved two to three times as often. With balancing on, fio's own
CPU time per I/O doubles here, from 3.1 to 6.3 us, against 3.4 to 4.9 us
with TCM/USER, spread over the whole submission path rather than in one
function.

Turning load balancing off for fio's CPUs and nothing else reverses the
result. Single job, 4k randread at iodepth 32 over tcm_loop, 6 s per run,
five alternating runs, median (min-max). fio runs in a cpuset of CPUs
4-31 and the server threads are pinned to CPUs 0-3; the cpuset is either
a normal member (balanced) or an isolated partition (not balanced):

fio CPUs 4-31, backstore bandwidth (MB/s) change
balanced, TCM/USER 439 (398-462)
balanced, uring 379 (371-388) -13.7%
not balanced, TCM/USER 662 (544-710)
not balanced, uring 760 (575-798) +14.8%

Booting the guest with idle=poll removes the gap as well: with fio free
to move, five runs give a median of 612 MB/s with TCM/USER and 627 MB/s
here.

That points at what each switch costs in a guest. An idle vCPU executes
HLT, which exits to the host and puts the vCPU thread to sleep; waking
it for a kworker, ksoftirqd or a migrated fio takes an IPI, a host
wakeup and a VM entry, and the host may run the thread on another
physical CPU, so the guest code then runs with cold caches. With
idle=poll the vCPUs spin instead, stay on their physical CPUs and take
the IPI without an exit. This backstore makes more of those switches
per I/O, so it pays more of that cost, and fio's slowdown spread over
the whole submission path looks like cold caches.

What is measured is that idle=poll removes the gap and that the gap
follows the switches and migrations. Which part of the cost is the exit
and wakeup latency and which part is the vCPU thread landing on another
physical CPU is inferred: I cannot trace the host side here. Waking an
idle CPU is much cheaper on bare metal, so I expect the gap to be
smaller there, but I have not measured it. idle=poll is only a
diagnostic, it burns a host CPU per idle vCPU.

[...]

> When userspace target becomes more efficient, I'd suggest you to think
> about why these applications can't be improved or refactored for the
> potential (bigger)perf requirement.

Agreed. Since this backstore only takes block commands, an application
that can move to it can just as well move to ublk, and one that needs
SCSI semantics in userspace is not served by it either.

Thanks,
Qiliang