Re: Unbounded growth of slab caches and how to shrink them

[Christoph Lameter]

On Wed, 29 Jun 2016, Nikolay Borisov wrote:

> I've observed a rather strange unbounded growth of the kmalloc-192
> slab cache:
>
> OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 711124869 411527215   3%    0.19K 16934908       42 135479264K kmalloc-192
>
> Essentially the kmalloc is around 130 GB , yet only 3 percent of this are
> being used. In this case I'd like to essentially shrink the overall size
> of the cache. How is it possible to achieve that? I tried echoing '1'
> to /sys/kernel/slab/kmalloc-192/shrink but nothing changed.

Ok this probably means that most slabs have just a few or one objects?
Some workloads can result in situations like that. Can you enable
debugging and get a list of functions where these objects are allocated?

> This is on 3.12 which is rather old kernel, but still I believe it is
> entirely possible for someone to find a way to flood a machine with
> network requests which would cause a lot of objects to be allocate,
> resulting in a particular slab cache growing, then later when the request
> flood stops the cache would be almost empty, yet the memory won't be usable
> for anything other than satisfying memory allocation from this cache.

True. Long known problem and all my attempts to facilitate a solution here
did not go anywhere. The essential solution would require objects being
movable or removable from the sparsely allocated page frames. And this
goes way beyond my subsystem.

If you can figure out which subsystem allocates or frees these objects
(through the call traces) then we may find a knob in the subsystem to
clear those out once in a while.