[REGRESSION] tmpfs: flistxattr(fd) small-list slowdown after simple_xattrs rhashtable conversion

[Chengfeng Lin]

Hi,

I found a small-list flistxattr(fd) slowdown on tmpfs while looking at FS
performance regressions.  This is a performance regression report.

The scope is intentionally narrow:

  tmpfs + user xattrs + flistxattr(fd) + small xattr lists

The cleanest reproducer creates a tmpfs file, sets one or more `user.*`
xattrs, and repeatedly measures:

  flistxattr(fd, list, list_size)

The single-xattr scenario checks that the returned list is non-empty; the
count-gradient scenario also checks the exact returned list length.

The bare-metal test machine is an x86-64 Intel Core i7-14700 system with 28
logical CPUs and 30 GiB RAM.  The benchmark was pinned to CPU 2, and the test
filesystem was tmpfs.

On bare metal, the parent/child A/B around:

  52b364fed6e1 shmem: adapt to rhashtable-based simple_xattrs with
lazy allocation

shows a clear jump.  The parent has the rhashtable infrastructure but tmpfs
still uses the old rbtree path.  The child switches tmpfs to the lazy
rhashtable-based simple_xattrs path.  I understand that the lazy allocation
part is meant to avoid per-inode rhashtable overhead for tmpfs inodes that do
not use xattrs; this report focuses on the small-list listxattr cost for tmpfs
inodes that already have user xattrs.

With one xattr, skipping the first round:

  kernel / state                         flistxattr_ns_per_op
  parent A, tmpfs old rbtree path         135.754
  child,  tmpfs lazy rhashtable path      229.663
  parent B, tmpfs old rbtree path         135.335

That is about +69.4% for the child vs the average of the two parent runs.
The two parent runs agree closely, and the skip-first-round ranges were:

  parent A: 134.451..140.153 ns/op
  child:   217.758..239.498 ns/op
  parent B: 134.273..138.098 ns/op

All of these runs reported `expected_match_ratio=100` and
`unexpected_results=0`.

This run used 15 rounds and 65536 iterations per target, pinned to one CPU.
The parent and child kernels were built with the same GCC 13.3 toolchain and
their `.config` files differ only in `CONFIG_LOCALVERSION`.
`CONFIG_TMPFS_XATTR` was enabled, `CONFIG_KASAN` was disabled, and the configs
had `CONFIG_DEBUG_KERNEL=y` / `CONFIG_SLUB_DEBUG=y`.

I also reran the same parent/child/parent A/B with `CONFIG_SLUB_DEBUG` and
`CONFIG_KASAN` disabled.  `CONFIG_DEBUG_KERNEL` remained enabled in this
config because it is selected by `CONFIG_EXPERT=y`, so I am not treating this
as a full `DEBUG_KERNEL=n` build.  In that matched production-like run,
skipping the first round:

  kernel / state                         flistxattr_ns_per_op
  parent A, tmpfs old rbtree path         148.536
  child,  tmpfs lazy rhashtable path      223.866
  parent B, tmpfs old rbtree path         149.178

So the child was still about +50.4% slower than the average of the two parent
runs (`223.866 ns/op` vs `148.857 ns/op`).  This is a smaller absolute delta
than the debug-config A/B, but it still shows the same direction after
removing the heavier SLUB/KASAN debug options.

I also ran a count gradient on the same parent/child pair:

  xattr_count   parent A ns/op   child ns/op   parent B ns/op   child vs avg
  1             156.102          242.886       156.872          +55.2%
  4             185.109          273.756       181.206          +49.5%
  16            312.894          378.211       315.962          +20.3%
  64            885.222          934.356       870.759          +6.4%

So this looks like a small-list fixed-cost regression rather than a claim that
rhashtable-based xattrs are slower for every workload or every list size.  As
the number of xattrs grows, the extra fixed cost is amortized by the list walk
itself.  These are tight-loop microbenchmark numbers, so the absolute per-op
cost is small; I am reporting it as a fixed-cost regression in this
enumeration path, not as measured impact on a specific application.  I am also
not claiming that setxattr/getxattr or non-tmpfs xattr workloads have the same
behavior.

Earlier attribution runs pointed in the same direction:

  - the signal narrowed to flistxattr(fd), not fsetxattr(fd) or fgetxattr(fd);
  - bpftrace layer timing did not point at security_inode_listxattr();
  - the growing part was around shmem_listxattr() / simple_xattr_list();
  - a diagnostic rbtree-backend probe recovered most of the old timing for this
    workload, although it changes the backend shape and is only attribution
    evidence.

I am not sending the diagnostic probe as a patch, because it is only
attribution evidence.  I wanted to ask whether this small-list tmpfs
flistxattr(fd) slowdown is expected/acceptable from the rhashtable conversion,
or whether a small-list fast path would be worth exploring without weakening
the original lazy-allocation goal.

Evidence bundle:

  https://github.com/lcf0399/linux-regression-evidence/tree/f0a07aa70b79219bd9f47f577357d673b3a06dae/tmpfs-flistxattr-small-list

Bare-metal summaries:

  https://github.com/lcf0399/linux-regression-evidence/tree/f0a07aa70b79219bd9f47f577357d673b3a06dae/tmpfs-flistxattr-small-list/bare-metal

Standalone reproducer:

  https://github.com/lcf0399/linux-regression-evidence/tree/f0a07aa70b79219bd9f47f577357d673b3a06dae/tmpfs-flistxattr-small-list/reproducer

Thanks,
Chengfeng