Re: [PATCH v2] vmstat: Keep count of the maximum page reached by the kernel stack

[Pasha Tatashin]

On Mon, Mar 18, 2024 at 4:40 PM Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> wrote:
>
> On Thu, 14 Mar 2024 14:54:57 +0000 Pasha Tatashin <pasha.tatashin@xxxxxxxxxx> wrote:
>
> > CONFIG_DEBUG_STACK_USAGE provides a mechanism to determine the minimum
> > amount of memory left in a stack. Every time a new low-memory record is
> > reached, a message is printed to the console.
> >
> > However, this doesn't reveal how many pages within each stack were
> > actually used. Introduce a mechanism that keeps count the number of
> > times each of the stack's pages were reached:
> >
> >       $ grep kstack /proc/vmstat
> >       kstack_page_1 19974
> >       kstack_page_2 94
> >       kstack_page_3 0
> >       kstack_page_4 0
> >
> > In the above example, out of 20,068 threads that exited on this
> > machine, only 94 reached the second page of their stack, and none
> > touched pages three or four.
> >
> > In fleet environments with millions of machines, this data can help
> > optimize kernel stack sizes.
>
> We really should have somewhere to document vmstat things.

We really should have a documentation for both procfs and sysfs
versions of these files:

$ wc -l /proc/vmstat
177 /proc/vmstat

$ wc -l /sys/devices/system/node/node0/vmstat
61 /sys/devices/system/node/node0/vmstat

Some of the counters are shared between the two (where procfs contains
machine global view), and some such as vm_event are only part of
/proc/vmstat. All of that requires a documentation somewhere under
Documentation/mm/vmstat.rst. We must explain that this is not a stable
API, as these counters depend on the kernel internal implementation.
However, there are so many of them, that it will take some effort to
do the initial explanation of all of them.

> > --- a/include/linux/vm_event_item.h
> > +++ b/include/linux/vm_event_item.h
> > @@ -153,10 +153,39 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
> >               VMA_LOCK_ABORT,
> >               VMA_LOCK_RETRY,
> >               VMA_LOCK_MISS,
> > +#endif
> > +#ifdef CONFIG_DEBUG_STACK_USAGE
> > +             KSTACK_PAGE_1,
> > +             KSTACK_PAGE_2,
> > +#if THREAD_SIZE >= (4 * PAGE_SIZE)
> > +             KSTACK_PAGE_3,
> > +             KSTACK_PAGE_4,
> > +#endif
> > +#if THREAD_SIZE > (4 * PAGE_SIZE)
> > +             KSTACK_PAGE_REST,
> > +#endif
> >  #endif
> >               NR_VM_EVENT_ITEMS
> >  };
>
> This seems a rather cumbersome way to produce a kind of histogram.  I
> wonder if there should be a separate pseudo file for this.

If you would like, the #if for stack size can be removed, I added them
not to print kstack_page_3 and kstack_page_4 on order 1 stacks where
there are only two-pages. This series shows the frequency of reaching
each of these pages by the existing threads.

> And there may be a call for extending this.  For example I can forsee
> people wanting to know "hey, which process did that", in which case

Which process did that (to some extent) would be what is printed out
by: CONFIG_DEBUG_STACK_USAGE when it finds the new record size stack.
Other than that, we could use tracing to find the callers that cause
these deep stacks.

However, the information provided by this patch can help to figure out
if tracing is needed or not. For example, if it is known that the
third or forth pages are extremely rarely used, say 0.00001% of time,
they could have a special API for deep stack, and save half of the
stack  memory in the fleet.

Thank you,
Pasha