Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger]

I reran the matrix. I'm at a loss for why, but I can no longer reproduce
the "bad" throughput with cache_shard. I saw that a few times last week.
Furthermore, the "bad" throughput is now a 10% degradation rather than 50%.
Nevertheless, it's still reliably reproducible when CQs all stack on NUMA 0
and using smt affinity_scope.

Patches 1-2 alone do not reproduce it across 5 trials where 3 of them fell
32/0 and 2 fell 17/15 on NUMA 0/1. All had full, unchanged throughput.

Patches 3-8 do reproduce it (as well as the full 1-8): throughput dips by
10% when all CQs fall on NUMA 0. Another behavior that I'm seeing again is,
within the same run, toggling affinity_scope to cache_shard fully restores
throughput, then back to smt and the throughput falls again.

The mpstat results, CPU idle times:

                        All NUMA 0          17/15 Split
    aggregate           20%                 70%
    CPU 0-23,48-71      0-2%                60-80%
    CPU 24-47,72-95     40-46%              70-90%

This shape is generally consistent among all three patch subsets: patches
1-2, patches 3-8, and patches 1-8. All of these are without modifying
affinity_scope for the respective defaults.

For patches 3-8 I reran the same experiment but captured mpstat on each
segment of smt -> cache_shard -> smt, on a run where all CQs were on NUMA
0, all idle times again:

                        smt         cache_shard     smt
    aggregate           15%         32%             19%
    CPU 0-23,48-71      0-1%        6-9%            0-2%
    CPU 24-47,72-95     28-32%      55-61%          35-40%

I've also seen all CQs land on NUMA 1, which seems less common than NUMA 0,
but the same behavior appears there with the node roles reversed.

Would there be a downside to using WQ_AFFN_CACHE_SHARD here instead of
WQ_AFFN_SMT?