Re: [RFC PATCH 3/3] io_uring/rsrc: prefill the node cache when a file table is registered empty

[Gabriel Krisman Bertazi]

Uzair Beg <uzairbeg11@xxxxxxxxx> writes:

> Gabriel Krisman Bertazi <krisman@xxxxxxx> writes:
>> Does it make
>> sense to pre-allocated up to 192KB in memory for short-lived
>> applications that might use only a couple of those nodes?
>
> As a default, I don't think it does. Would it be acceptable to drop the
> prefill and only let the node cache capacity follow the table size,
> capped?
>
> Nothing would be allocated up front beyond the pointer array,
> so registering a large table and using a few slots costs almost
> nothing, and nodes are only retained up to what the application
> actually had installed. That gives up the first fill result but keeps
> the churn case, and it needs neither the dedicated slab nor the bulk
> refill.
>
> I'll measure that variant and follow up with numbers before posting
> anything. If you would rather the node cache stay at a fixed size,
> that is useful to know too.

The problem with increasing the cache size is that we don't have (and
should not have) a reclaim mechanism for the cached nodes.  By
increasing the cache size arbitrarily for every ring in the system you
are now sitting on a much bigger pile of unreclaimable memory that most
applications are unlikely to need.  I think that would even be
worse as a general solution.

Beyond microbenchmarks, this would only be a "problem" for applications
with a working set that frequently recycles over 128 nodes at a time.
The question remains how well this microbenchmark replicates any
real-world scenarios.  And if 128 is not enough, what would be? To be
fair, this is true for all type of magazine-like caches we have in
io_uring. In addition, you install a fd because you want to do one or
more operations with it, diluting the update cost.  In this case, would
the cost of going to slab for a node update of a working set of >128 fds
be diluted by the other operations, and likely invisible?  If it is
really a problem, I guess I would be ok with the bulk refilling of the
128 nodes at a time when you have an empty cache missing, while keeping
the table the same size.

On another topic... it would be excellent if we could do bulk refill on the
shared slab without a separate a kmem_cache!

Thanks,

-- 
Gabriel Krisman Bertazi