Re: [PATCH 0/3] further damage-control lack of clone scalability

[Mateusz Guzik]

On Wed, Dec 3, 2025 at 9:37 AM Mateusz Guzik <mjguzik@xxxxxxxxx> wrote:
>
> On Sun, Nov 23, 2025 at 4:00 PM Matthew Wilcox <willy@xxxxxxxxxxxxx> wrote:
> >
> > On Sun, Nov 23, 2025 at 07:30:51AM +0100, Mateusz Guzik wrote:
> > > When spawning and killing threads in separate processes in parallel the
> > > primary bottleneck on the stock kernel is pidmap_lock, largely because
> > > of a back-to-back acquire in the common case.
> > >
> > > Benchmark code at the end.
> > >
> > > With this patchset alloc_pid() only takes the lock once and consequently
> > > alleviates the problem. While scalability improves, the lock remains the
> > > primary bottleneck by a large margin.
> > >
> > > I believe idr is a poor choice for the task at hand to begin with, but
> > > sorting out that out beyond the scope of this patchset. At the same time
> > > any replacement would be best evaluated against a state where the
> > > above relock problem is fixed.
> >
> > Good news!  The IDR is deprecated.  Bad news!  I'm not 100% sure that
> > the XArray is quite appropriate for this usecase.  I am opposed to
> > introducing more IDR APIs.  Have you looked at converting to the XArray?
> > Or do you have a better data structure in mind than the XArray?
> >
>
> Hi Willy,
>
> in other responses I outlined what I suspect would be a viable long
> term solution, very much not idr-based.
>
> However, that's not something I'm likely to implement anytime soon and
> I doubt there is someone willing to pick up the matter.
>
> Whatever the long term solution and who/when implements it, the
> current code avoidably loses out on some performance because of at
> least two lock acquires on each fork instead of one.
>
> At the moment it is structured in a way which makes it possible to
> take the lock once with minor effort.
>
> Leaving this in the current state results in a minor risk that someone
> will make changes which turn fixing the scalability issue into a
> massive problem.
>
> With my patch as-is I can suffer some pain and avoid modifying
> idr_prealloc, but at the same time I don't think the modification at
> hand is particularly problematic. Notably it does not change any of
> the internals.
>
> So the question is if by "opposed to introducing more IDR APIs" you
> mean you are just not fond of it, or is it a straight up NAK. Per the
> explanation above, I think the change is tolerable in its own right
> and I provided reasoning why I'm adding it in the first place.
>
> If the latter, I'll see about massaging this to drop locks and retry
> memory alloc.

Welp, retrying preload turned out to be significantly less painful
than I thought, so you can consider the above question moot. I'll be
posting v2 without idr changes.