Re: [RFC PATCH] fsnotify: raise default fs.inotify.max_user_instances now that memcg accounts for it

[Harsh Upadhayay]

Hi Jan and Amir,

Thank you both for the detailed feedback and for explaining the
perspective on keeping the default static. I appreciate the
clarification regarding the risks of unintended DoS and the
architectural considerations for nested environments.

I understand that this is better handled as an administrative
configuration for specific workloads like Kubernetes rather than a
change to the upstream default. I'll drop the proposal.

Best regards,

Harsh

On Wed, Aug 19, 2026 at 5:23 AM Amir Goldstein <amir73il@xxxxxxxxx> wrote:
>
> On Tue, Aug 18, 2026 at 3:47 PM Jan Kara <jack@xxxxxxx> wrote:
> >
> > Hello!
> >
> > On Wed 12-08-26 20:51:36, Harsh Upadhayay wrote:
> > > I hit this running a 3-node k3s homelab cluster: every container on a
> > > node shares one 128-slot fs.inotify.max_user_instances pool (no user
> > > namespace remapping), kubelet alone was holding 25-40+ instances just
> > > from ConfigMap/Secret volume watching, and the pool emptied under
> > > normal load -- surfaced as an unhelpful, inotify-silent "too many open
> > > files" in whichever container's turn it was to ask next. Filing as
> > > RFC since I couldn't find prior discussion of raising this default.
> >
> > Not sure what Amir thinks about this but I think default is just a default
> > and it is natural it is not good for everybody. I can see how 128 inotify
> > instances may not be enough for Kubernetes setups but then the answer is
> > for Kubernetes installation to configure maximum inotify instances to a
> > higher value. And as you show below, different container runtimes tune the
> > value differently which just proves my point that whatever value we choose
> > people will need to tune it - which is the point of this limit being tunable
> > after all.
>
> I agree that I don't see a strong enough reason to change an upstream default
> to another arbitrarily bigger value that could be easily changed by the distro
> or by the kubernetes host.
>
> >
> > So I don't see a strong enough reason to bump up the default limit because
> > there can be other users for which increased limit could allow a potential
> > DoS in their setups. The concern isn't as much the memory consumed by
> > inotify instances themselves as you write below but more all the events
> > that can get queued to each instance. OTOH those are accounted to the
> > instance's memcg so your overall argument that in memcg enabled setups the
> > limits aren't that much needed is still somewhat valid.
> >
>
> Not only the memory of marks nor events but also processing time and
> arbitrary long delays to other users in the system.
> I'll explain this with an extreme example below...
>
> > > fs.inotify.max_user_instances has been a flat 128 since inotify's
> > > introduction in 2005. That number is not per-container or per-workload
> > > -- it is a single pool shared by every process running under a given
> > > real UID on the host. On a multi-tenant Linux box, that means every
> > > container's root process shares one 128-slot pool with every other
> > > unremapped container's root, plus the host's own kubelet/containerd
> > > (or equivalent) bookkeeping, since user-namespace UID remapping is
> > > off by default in essentially every mainstream container runtime.
> > >
>
> ...let's do a thought experiment, why stop at one multi-tenant level
> why not nest it several times as userns is designed to do to allow for
> containers inside containers.
>
> In this hypothetical environment I am not sure it makes sense for
> users in container A to pay for the cost of subscribers in nested
> containers B.*.*.
>
> It may make more sense in nested environment for container B to have a
> single group in container B getting the events for all the subscribers in B.*
> and forwarding the events to sub-subscribers and so forth in B.*.*.
>
> Maybe the group in container B does not have inode marks but fs marks
> with bpf inode maps as filter.
>
> IOW, if kubernetes needs to increase max_user_instances for modern day
> workloads - it may do that, but I don't think this design extends well into
> a future of many more containers on the same host.
> Even if userns were used by kubernetes, I am not sure that the current
> publish subscribe model of inode marks would be able to handle thousands
> of userns well.
>
> Thanks,
> Amir.