Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[tmenninger]

Sorry for the delay, I've been without internet.

> 1. Check that sheaves are actually on for that cache:
>
>       cat /sys/kernel/slab/nfs_page/sheaf_capacity
>
>    A zero means your kernel predates sheaves or the cache has debug
>    flags, and the contention is the older remote-free slab
>    transition path. That is a different report.

    root@irp309-h02:/sys/kernel/slab/nfs_page# cat sheaf_capacity
    60

> 2. Build with CONFIG_SLUB_STATS=y and sample these before and after
>    a 10-second all-local window:
>
>    ```
>    cd /sys/kernel/slab/nfs_page
>    grep . free_fastpath free_slowpath sheaf_flush barn_put \
>        barn_put_fail barn_get barn_get_fail alloc_slowpath
>    ```
>
>    If barn_put_fail and sheaf_flush track the contention count, the
>    barn limit is the bottleneck. If free_slowpath dominates and the
>    sheaf counters are quiet, the frees are bypassing sheaves
>    entirely and my analysis is wrong.

10-second deltas:

    free_fastpath:    +49,161,373
    free_slowpath:    +70,391,872
    sheaf_flush:       +1,582,440
    barn_put:            +673,877
    barn_put_fail:        +26,374
    barn_get:            +673,860
    barn_get_fail:     +1,199,576
    alloc_slowpath:             0

> 3. Run perf lock contention with -l on the same window so the lock
>    shows up by address, and note whether the callers under
>    list_lock are sheaf_flush_unused / __kmem_cache_free_bulk or
>    direct __slab_free from kmem_cache_free.

    contended       total wait     avg wait      address
    3,346,091       5.05 min       90.48 us      ff1db88b8f578b80

Filtering the perf lock data to that address gives:

    2,845,710   __slab_free+0x126
      318,458   get_partial_node_bulk+0x9c
      181,842   __refill_objects_node+0x4d0
           71   alloc_from_new_slab+0x183
           10   __slab_free+0x126

The next most contended lock had only 23 contentions.

So about 85% of the contention on the dominant lock is coming directly
from __slab_free. I don't see sheaf_flush_unused or
__kmem_cache_free_bulk among the callers. get_partial_node_bulk is
about 9.5% and __refill_objects_node about 5.4%.

That seems to put this much more firmly in the direct
kmem_cache_free -> __slab_free case you mentioned rather than a barn
limit / sheaf flush bottleneck.

> 4. Tell me the I/O pattern: direct reads, I assume, but the I/O
>    size, number of threads, and how they are spread across the two
>    nodes. I would like to attempt to reproduce the
>    alloc-here-free-there imbalance rather than speculate.

Yes, direct reads. The workload is:

    elbencho \
        --iter 1 \
        --threads 160 \
        --files 16 \
        --size 512G \
        --block 1m \
        --dropcache \
        --iodepth 64 \
        --direct \
        --read \
        --lat \
        --latpercent \
        --log 1 \
        <16 files>

I'm not pinning the elbencho threads to CPUs or NUMA nodes. I just waited
for a run where all of the CQs landed on node 0.