[PATCH 0/4] kernfs: remove kernfs_rwsem from dentry revalidation

[Shakeel Butt]

At Meta, we are seeing important system daemons that poll cgroupfs and
sysfs geth stuck in kernfs_dop_revalidate() for minutes. The two that
hurt most are the ones we can least afford to lose: oomd, which decides
what to kill when a machine runs out of memory, and below[1], which
records the telemetry used to understand what happened afterwards.

kernfs_dop_revalidate() takes kernfs_rwsem for read once per path component
of every walk into a kernfs mount. Linux rwsems do not permit reader lock
stealing once a writer is queued, so one writer -- a cgroup created or
destroyed, a device renamed -- parks the entire incoming reader stream in
uninterruptible sleep. Daemons polling cgroup files in a loop are exactly
the workload that turns this into a convoy, and cgroup churn is exactly
what a busy machine does.

Nothing the callback reads needs the semaphore. kn->active is an atomic_t
already tested lock-free elsewhere, kn->__parent and kn->name are RCU
pointers, kn->ns can be compared rather than dereferenced, and
parent->dir.rev is a plain counter.

  1/4 uses the parent inode and name the VFS already passes to
      ->d_revalidate() rather than recovering them from mutable dentry
      fields, comparing the name by explicit length
  2/4 annotates the directory revision counter for lockless access
  3/4 compares namespace tags by pointer
  4/4 removes kernfs_rwsem from the callback

LOOKUP_RCU still returns -ECHILD. kernfs_iop_permission() forces every walk
out of RCU-walk before children are revalidated, so lifting it here would
have no effect until that path is fixed; left to a separate series.

Readers walking cgroupfs and sysfs while another thread churns cgroups,
renames netdevs and adds/removes devices, on an 8-CPU VM:

                    kernfs_rwsem read   contentions   revalidate among
                        acquisitions                  top call sites
  before                  48,593,360     1,744,517    #1 and #2
  after                    1,280,280       429,846    absent

Reader path-walk throughput improved 35-53% over the same workload.

Tested against an unpatched control of the same tree, built and booted with
KASAN, KCSAN (default and STRICT), PROVE_LOCKING, PROVE_RCU,
DEBUG_ATOMIC_SLEEP and LOCK_STAT. The deactivated, renamed and
namespace-moved reject paths and negative-dentry invalidation all behave as
before. KCSAN_STRICT over 180s reports no data race involving
kernfs_dop_revalidate() or any field it reads, and there are no KASAN,
lockdep or might-sleep reports across millions of concurrent path walks.
The only kernfs KCSAN reports are in kernfs_refresh_inode(), present
identically on the control and addressed separately.

Link: https://github.com/facebookincubator/below [1]

Shakeel Butt (4):
  kernfs: Use VFS lookup context in d_revalidate()
  kernfs: Prepare directory revisions for lockless reads
  kernfs: Avoid namespace dereference in d_revalidate()
  kernfs: Remove kernfs_rwsem from dentry revalidation

 fs/kernfs/dir.c             | 76 +++++++++++++++----------------------
 fs/kernfs/kernfs-internal.h |  9 ++---
 2 files changed, 35 insertions(+), 50 deletions(-)


base-commit: 7079a12d7506b07fb53b54a664bfad5fa9b16d70
-- 
2.53.0-Meta