Re: Re: Re: [PATCHSET wq/for-6.8] workqueue: Implement system-wide max_active for unbound workqueues

[Naohiro Aota]

Hello,

I'm sorry for the late response.

On Tue, Jan 16, 2024 at 11:04:53AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Mon, Jan 15, 2024 at 05:46:07AM +0000, Naohiro Aota wrote:
> > CPU: Intel(R) Xeon(R) Platinum 8260 CPU, 96 cores
> > NUMA nodes: 2
> > RAM: 1024 GB
> > 
> > However, for another benchmark experiment I'm doing, I booted the machine
> > with "numa=off mem=16G" kernel command-line. I admit this is an unusual
> > setup...
> 
> So, does that end up using only memory from one node while making the kernel
> unaware of NUMA topology?

Yes.

> 
> > On that machine, I create a fresh btrfs with "mkfs.btrfs -d raid0 -m raid0
> > <devices>" with 6 SSD devices. And, I run the following command on the FS.
> > 
> > fio --group_reporting --eta=always --eta-interval=30s --eta-newline=30s \
> >     --rw=write --fallocate=none \
> >     --direct=1 --ioengine=libaio --iodepth=32 \
> >     --filesize=100G \
> >     --blocksize=64k \
> >     --time_based --runtime=300s \
> >     --end_fsync=1 \
> >     --directory=${MNT} \
> >     --name=writer --numjobs=32
> > 
> > tools/workqueue/wq_dump.py output is pasted at the
> > bottom. "btrfs-endio-write" is the workqueue, which had many workers on the
> > unpatched kernel.
> 
> If so, I'm not sure how meaningful the result is. e.g. The perf would depend
> heavily on random factors like which threads end up on which node and so on.
> Sure, if we're slow because we're creating huge number of concurrent
> workers, that's still a problem but comparing relatively small perf delta
> might not be all that meaningful. How much is the result variance in that
> setup?

Yeah, that is true. I conducted the benchmark 30 times, and the sample standard
deviation is 320.30. They ranged as follow.
Min 1732 MiB/s - Max 2565 MiB/s
Mean: 2212.3 MiB/s Sample stddev 320.30

Comparing to that, here is the result on the baseline.
Min 1113 MiB/s - Max 1498 MiB/s
Mean: 1231.85 Sample stddev 104.31

For a reference, a result on reverted case is as follow:
Min 2211 MiB/s - Max 2506 MiB/s
Mean 2372.23 MiB/s Sample stddev 82.49

So, the patched one is indeed better than the baseline. Even the worst case
on patched version is better than the best on baseline. And, as you
mentioned. patched version has far larger variance than baseline and
reverted one.

> 
> > FYI, without the kernel command-line (i.e, numa=on and all RAM available as
> > usual), as shown below, your patch series (v1) improved the performance
> > significantly. It is even better than the reverted case.
> > 
> > - misc-next, numa=on
> >   WRITE: bw=1121MiB/s (1175MB/s), 1121MiB/s-1121MiB/s (1175MB/s-1175MB/s), io=332GiB (356GB), run=303030-303030msec
> > - misc-next+wq patches, numa=on
> >   WRITE: bw=2185MiB/s (2291MB/s), 2185MiB/s-2185MiB/s (2291MB/s-2291MB/s), io=667GiB (717GB), run=312806-312806msec
> > - misc-next+wq reverted, numa=on
> >   WRITE: bw=1557MiB/s (1633MB/s), 1557MiB/s-1557MiB/s (1633MB/s-1633MB/s), io=659GiB (708GB), run=433426-433426msec
> 
> That looks pretty good, right?

Yes, it is so good. Since the numa=off case is quite unusual and it has a
large variance, I believe this patch series is a good improvement.

> 
> Thanks.
> 
> -- 
> tejun