Re: [PATCH v4] kernfs: allocate inodes and dentry outside kernfs_rwsem

From: Alexander Mikhalitsyn

Date: Thu Nov 19 2026 - 07:32:49 EST


On Wed, 2026-11-11 at 13:50 -0800, Shakeel Butt wrote:
> In the Meta fleet, on 6.16 with the kernfs changes from 7.4
> backported,
> we still see long cgroupfs stalls:
>
> - oomd and below waited 72 seconds in kernfs_fop_readdir() while an
> out
>   of memory container created cgroups.  They resumed when the kernel
> OOM
>   killer killed the container's workload.
>
> - oomd waited 15 minutes in kernfs_iop_lookup().  PID 1 waited for
>   kernfs_rwsem in cgroup_rmdir() while holding cgroup_mutex.
>
> - A lock profiler sampling part of the fleet saw kernfs_iop_lookup()
>   hold kernfs_rwsem across memcg reclaim on 26 hosts in 29 days, for
> up
>   to 625 ms.
>
> The profiler shows where these stalls come from.  kernfs_iop_lookup()
> and the export ops allocate a new inode while holding kernfs_rwsem
> for
> read.  The inode is charged to the caller's memcg.  If that memcg is
> at
> memory.max, the allocation can stay in reclaim for a long time,
> sometimes until an OOM kill.  A writer then queues on the lock and
> every
> new reader queues behind it.  On cgroup2 this blocks every cgroupfs
> user
> on the machine.
>
> Such lookups are common in a memcg that is out of memory.  Its
> reclaim
> drops its kernfs dentries fast, and the files of a new cgroup have no
> inodes yet.
>
> Allocate the inode before taking kernfs_rwsem, as kernfs_iop_mkdir()
> already does, and attach it under the lock with
> kernfs_get_inode_locked().  That helper adds a new inode only for an
> active node, so __kernfs_remove() still finds every inode it must
> clear
> i_nlink on, as commit dc1a0cc95737 ("kernfs: don't lose
> IN_DELETE_SELF
> when decoding a file handle") needs.
>
> The inode goes unused when the name does not exist, the node is
> inactive, or it already has an inode.  In a VM, a lookup of a missing
> name takes about 1 us longer, 4.5 us instead of 3.5 us on cgroupfs.
> Cold lookups of existing sysfs and cgroupfs files take 2 to 3%
> longer.
>
> kernfs_get_parent_dentry() also held the lock across
> d_obtain_alias(),
> which allocates a dentry.  Hold it only around the inode lookup, as
> __kernfs_fh_to_dentry() already does.  This also gives ->get_parent
> the
> active test that the IN_DELETE_SELF fix left out: a parent that is
> being
> removed now gets -ESTALE.  Before, it got an inode anyway, possibly a
> new one whose i_nlink __kernfs_remove() never cleared.
>
> kernfs_fill_super() preallocates the root inode the same way, so no
> inode is allocated under kernfs_rwsem any more.
>
> Tested on a 316-CPU machine.  A task in a full 256 MB memcg with slow
> swap looks up the files of new cgroups.  Worst waits in 60 seconds:
>
>                                   before    after
>   oomd, for kernfs_rwsem          236 ms   0.1 ms
>   below, for kernfs_rwsem         356 ms   0.8 ms
>   systemd, for kernfs_rwsem       160 ms    37 us
>   readdir of another directory    530 ms   3.9 ms
>
> The lookup itself is still slow.  The VFS still holds its directory's
> lock, so other work in that one directory still waits.
>
> Suggested-by: Tejun Heo <tj@xxxxxxxxxx>
> Acked-by: Tejun Heo <tj@xxxxxxxxxx>
> Signed-off-by: Shakeel Butt <shakeel.butt@xxxxxxxxx>

Hi Shakeel,

please feel free to add

Reviewed-by: Alexander Mikhalitsyn <alex@xxxxxxxxxxxx>

Kind regards,
Alex

> ---
>
> Christian, this depends on Jori's series in vfs-7.5.directory.  Could
> you
> take it on top of it?  Greg, you asked to wait for -rc1 and for
> testing
> in -next.  Are you OK with this going through the vfs tree?
>
> Changes since v3:
> - Rebased on vfs-7.5.directory.  Use the kernfs_get_inode_locked()
> that
>   Jori added there for kernfs_iop_mkdir().
> - ->get_parent: a parent that is being removed now gets -ESTALE even
> if
>   it already has an inode, because the helper tests the node first.
> - New cost numbers. v3 costs the same as v4 in it.
> - Kept Tejun's ack, as the approach he suggested is unchanged.
>
> v3:
> https://lore.kernel.org/20261016220029.3276154-1-shakeel.butt@xxxxxxxxx
>
> Changes since v2:
> - Allocate the inode before taking kernfs_rwsem and free it if
> unused,
>   instead of dropping and retaking the lock.  kernfs_iget() becomes
>   kernfs_get_inode_locked() (Tejun).
> - Do the same in kernfs_fill_super() (Tejun).
> - Describe the ->get_parent active test (Tejun).
> - Retested on another machine, numbers updated.
>
> v2:
> https://lore.kernel.org/20261016164345.31618-1-shakeel.butt@xxxxxxxxx
>
> Changes since v1:
> - dentry allocation outside kernfs_rwsem (Sashiko).
> - Couple of comments updated.
>
> v1:
> https://lore.kernel.org/20261015221110.3183461-1-shakeel.butt@xxxxxxxxx
>
>  fs/kernfs/dir.c   | 12 ++++++++++--
>  fs/kernfs/inode.c |  3 +++
>  fs/kernfs/mount.c | 49 +++++++++++++++++++++++++++++----------------
> --
>  3 files changed, 43 insertions(+), 21 deletions(-)
>
> diff --git a/fs/kernfs/dir.c b/fs/kernfs/dir.c
> index e72783843edb..fb1dcf35d900 100644
> --- a/fs/kernfs/dir.c
> +++ b/fs/kernfs/dir.c
> @@ -1251,9 +1251,12 @@ static struct dentry *kernfs_iop_lookup(struct
> inode *dir,
>   struct kernfs_node *parent = dir->i_private;
>   struct kernfs_node *kn;
>   struct kernfs_root *root;
> - struct inode *inode = NULL;
> + struct inode *inode = NULL, *new;
>   const struct ns_common *ns = NULL;
>  
> + /* allocate outside kernfs_rwsem, see
> kernfs_get_inode_locked() */
> + new = new_inode(dir->i_sb);
> +
>   root = kernfs_root(parent);
>   down_read(&root->kernfs_rwsem);
>   if (kernfs_ns_enabled(parent))
> @@ -1267,9 +1270,13 @@ static struct dentry *kernfs_iop_lookup(struct
> inode *dir,
>   */
>   if (!kernfs_active(kn)) {
>   up_read(&root->kernfs_rwsem);
> + iput(new);
>   return NULL;
>   }
> - inode = kernfs_get_inode(dir->i_sb, kn);
> + if (new) {
> + inode = kernfs_get_inode_locked(new, kn);
> + new = NULL;
> + }
>   if (!inode)
>   inode = ERR_PTR(-ENOMEM);
>   }
> @@ -1282,6 +1289,7 @@ static struct dentry *kernfs_iop_lookup(struct
> inode *dir,
>   if (!IS_ERR(inode))
>   kernfs_set_rev(parent, dentry);
>   up_read(&root->kernfs_rwsem);
> + iput(new);
>  
>   /* instantiate and hash (possibly negative) dentry */
>   return d_splice_alias(inode, dentry);
> diff --git a/fs/kernfs/inode.c b/fs/kernfs/inode.c
> index 9cc1a56740b9..53b1a2796e0b 100644
> --- a/fs/kernfs/inode.c
> +++ b/fs/kernfs/inode.c
> @@ -269,6 +269,9 @@ static int kernfs_inode_test(struct inode *inode,
> void *data)
>   * while the inode is inserted and __kernfs_remove() always finds it
> to
>   * clear its i_nlink.  @inode is consumed.
>   *
> + * The caller allocates @inode before taking kernfs_rwsem, because
> the
> + * allocation can block in reclaim and hold up other users of the
> lock.
> + *
>   * Returns the inode now attached to @kn, which is @inode unless one
> was
>   * already hashed for it (e.g. by a file handle decode), or NULL if
> @kn
>   * is no longer active.
> diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c
> index a0be784bfb06..05e8ec1ee7fa 100644
> --- a/fs/kernfs/mount.c
> +++ b/fs/kernfs/mount.c
> @@ -99,8 +99,8 @@ static struct dentry *__kernfs_fh_to_dentry(struct
> super_block *sb,
>       int fh_type, bool
> get_parent)
>  {
>   struct kernfs_super_info *info = kernfs_info(sb);
> + struct inode *inode = ERR_PTR(-ESTALE), *new;
>   struct kernfs_node *kn;
> - struct inode *inode;
>   u64 id;
>  
>   if (fh_len < 2)
> @@ -124,32 +124,32 @@ static struct dentry
> *__kernfs_fh_to_dentry(struct super_block *sb,
>   return NULL;
>   }
>  
> + new = new_inode(sb);
> + if (!new)
> + return ERR_PTR(-ENOMEM);
> +
>   /*
> - * Hold kernfs_rwsem across the lookup as well as
> kernfs_get_inode().
> - * __kernfs_remove() deactivates the subtree and clears
> i_nlink on its
> - * inodes under the write lock, so under the read lock
> either
> - * kernfs_find_and_get_node_by_id() refuses the node, or the
> inode is
> - * in the inode hash before the ilookup() pass goes looking
> for it.
> + * Hold kernfs_rwsem from the lookup through
> kernfs_get_inode_locked().
> + * Then __kernfs_remove() either finds the inode to clear
> i_nlink, or
> + * the node is already inactive and no inode is created.
>   */
>   scoped_guard(rwsem_read, &info->root->kernfs_rwsem) {
>   kn = kernfs_find_and_get_node_by_id(info->root, id);
> - if (!kn)
> - return ERR_PTR(-ESTALE);
> -
> - if (get_parent) {
> + if (kn && get_parent) {
>   struct kernfs_node *parent;
>  
>   parent = kernfs_get_parent(kn);
>   kernfs_put(kn);
>   kn = parent;
> - if (!kn)
> - return ERR_PTR(-ESTALE);
>   }
> -
> - inode = kernfs_get_inode(sb, kn);
> - kernfs_put(kn);
> + if (kn) {
> + inode = kernfs_get_inode_locked(new, kn);
> + new = NULL;
> + kernfs_put(kn);
> + }
>   }
>  
> + iput(new);
>   return d_obtain_alias(inode);
>  }
>  
> @@ -171,9 +171,17 @@ static struct dentry
> *kernfs_get_parent_dentry(struct dentry *child)
>  {
>   struct kernfs_node *kn = kernfs_dentry_node(child);
>   struct kernfs_root *root = kernfs_root(kn);
> + struct inode *inode, *new;
>  
> - guard(rwsem_read)(&root->kernfs_rwsem);
> - return d_obtain_alias(kernfs_get_inode(child->d_sb,
> kernfs_parent(kn)));
> + new = new_inode(child->d_sb);
> + if (!new)
> + return ERR_PTR(-ENOMEM);
> +
> + scoped_guard(rwsem_read, &root->kernfs_rwsem) {
> + inode = kernfs_get_inode_locked(new,
> kernfs_parent(kn));
> + }
> +
> + return d_obtain_alias(inode);
>  }
>  
>  static const struct export_operations kernfs_export_ops = {
> @@ -294,7 +302,7 @@ static int kernfs_fill_super(struct super_block
> *sb, struct kernfs_fs_context *k
>  {
>   struct kernfs_super_info *info = kernfs_info(sb);
>   struct kernfs_root *kf_root = kfc->root;
> - struct inode *inode;
> + struct inode *inode, *new;
>   struct dentry *root;
>  
>   info->sb = sb;
> @@ -314,8 +322,11 @@ static int kernfs_fill_super(struct super_block
> *sb, struct kernfs_fs_context *k
>   sb->s_shrink->seeks = 0;
>  
>   /* get root inode, initialize and unlock it */
> + new = new_inode(sb);
> + if (!new)
> + return -ENOMEM;
>   down_read(&kf_root->kernfs_rwsem);
> - inode = kernfs_get_inode(sb, info->root->kn);
> + inode = kernfs_get_inode_locked(new, info->root->kn);
>   up_read(&kf_root->kernfs_rwsem);
>   if (!inode) {
>   pr_debug("kernfs: could not get root inode\n");
>
> base-commit: ecdfc51a6a78aaf1c4ac1d8b6ea199cc2ed020ac