[RFC PATCH] fsnotify: raise default fs.inotify.max_user_instances now that memcg accounts for it

[Harsh Upadhayay]

Hi all,

I hit this running a 3-node k3s homelab cluster: every container on a
node shares one 128-slot fs.inotify.max_user_instances pool (no user
namespace remapping), kubelet alone was holding 25-40+ instances just
from ConfigMap/Secret volume watching, and the pool emptied under
normal load -- surfaced as an unhelpful, inotify-silent "too many open
files" in whichever container's turn it was to ask next. Filing as
RFC since I couldn't find prior discussion of raising this default.

---

fs.inotify.max_user_instances has been a flat 128 since inotify's
introduction in 2005. That number is not per-container or per-workload
-- it is a single pool shared by every process running under a given
real UID on the host. On a multi-tenant Linux box, that means every
container's root process shares one 128-slot pool with every other
unremapped container's root, plus the host's own kubelet/containerd
(or equivalent) bookkeeping, since user-namespace UID remapping is
off by default in essentially every mainstream container runtime.

In practice this pool empties well before 128 "real" watchers are
running, because kubelet alone routinely holds 25-40+ instances on a
modestly loaded node just from tracking projected ConfigMap/Secret
volumes, before any application container asks for one. Once the pool
is empty, the next process on the node to call inotify_init() anywhere
gets EMFILE, surfaced by most userspace libraries as a generic "too
many open files" -- with no mention of inotify at all, making the
underlying cause hard to diagnose from the error text alone.

This is not a new observation. It has been independently hit and
worked around downstream, at the deployment-tooling layer rather than
the kernel default, at least by:
- Red Hat/OpenStack (dnsmasq-per-subnet exhausting the pool at scale;
fixed only in tripleo-heat-templates, defaulting to 1024) [1]
- NixOS containers (~20 containers sufficient to exhaust it) [2]
- Docker, which raises its own daemon default to 8192 [3]

Two ways to address this at the source were considered:

(A) Scale max_user_instances with available memory, mirroring the
formula already used for max_user_watches (1% of addressable
memory, clamped to [8192, 1048576], added in commit 928901237497
("inotify: Increase default inotify.max_user_watches limit to
1048576")). This keeps the two related limits symmetric and
self-tuning across wildly different machine sizes.

(B) Simply raise the static default, on the grounds that the
original justification for a low global ceiling -- guarding
against a single runaway process exhausting kernel memory via
unbounded inotify_init() calls -- has been substantially
addressed since commit ac7b79fd190b ("inotify, memcg: account
inotify instances to kmemcg", merged in 5.12), which charges
each instance's kernel memory to the creating process's memcg.
On any kernel >= 5.12, a workload already confined by a memory
cgroup (true of essentially every Kubernetes pod with resource
limits set) is already bounded by its own memory ceiling, not
by the global instance count. The global number no longer needs
to double as the primary abuse guard it was designed to be in
2005.

I'd lean towards (B): it's the smaller change, requires no new logic,
and leans entirely on a containment mechanism the kernel already
ships. It also has multi-year production precedent: Red Hat's
OpenStack fix has run with max_user_instances=1024 in production
environments since 2017 [1] without reported ill effects, and each
instance's fixed kernel-memory cost means the absolute increase (128
-> 1024 instances) is negligible even on small/embedded systems.

(A) remains a reasonable alternative or follow-up if maintainers would
rather keep instances and watches symmetric and self-tuning rather
than static -- happy to prepare that version instead if preferred.

Proposed diff, fs/notify/inotify/inotify_user.c, inotify_user_setup()
(current line 855 as of v6.x master):

- init_user_ns.ucount_max[UCOUNT_INOTIFY_INSTANCES] = 128;
+ init_user_ns.ucount_max[UCOUNT_INOTIFY_INSTANCES] = 1024;

[1] https://www.google.com/url?q=https://bugzilla.redhat.com/show_bug.cgi?id%3D1508030&source=gmail&ust=1786621798113000&sa=E
[2] https://www.google.com/url?q=https://github.com/NixOS/nixpkgs/issues/36214&source=gmail&ust=1786621798113000&sa=E
[3] Docker daemon default, inotify.max_user_instances=8192

Signed-off-by: Harsh Upadhayay <harshupadhayay906@xxxxxxxxx>