[PATCH 00/10] OOM Debug print selection and additional information

[Edward Chron]

This patch series provides code that works as a debug option through
debugfs to provide additional controls to limit how much information
gets printed when an OOM event occurs and or optionally print additional
information about slab usage, vmalloc allocations, user process memory
usage, the number of processes / tasks and some summary information
about these tasks (number runable, i/o wait), system information
(#CPUs, Kernel Version and other useful state of the system),
ARP and ND Cache entry information.

Linux OOM can optionally provide a lot of information, what's missing?
----------------------------------------------------------------------
Linux provides a variety of detailed information when an OOM event occurs
but has limited options to control how much output is produced. The
system related information is produced unconditionally and limited per
user process information is produced as a default enabled option. The
per user process information may be disabled.

Slab usage information was recently added and is output only if slab
usage exceeds user memory usage.

Many OOM events are due to user application memory usage sometimes in
combination with the use of kernel resource usage that exceeds what is
expected memory usage. Detailed information about how memory was being
used when the event occurred may be required to identify the root cause
of the OOM event.

However, some environments are very large and printing all of the
information about processes, slabs and or vmalloc allocations may
not be feasible. For other environments printing as much information
about these as possible may be needed to root cause OOM events.

Extensibility using OOM debug options
-------------------------------------
What is needed is an extensible system to optionally configure
debug options as needed and to then dynamically enable and disable
them. Also for options that produce multiple lines of entry based
output, to configure which entries to print based on how much
memory they use (or optionally all the entries).

Limiting print entry output based on object size
------------------------------------------------
To limit output, a fixed size of object could be used such as:
vmallocs that use more than 1MB, slabs that are using more than
512KB, processes using 16MB or more of memory. Such an apporach
is quite reasonable.

Using OOM's memory metrics to limit printing based on entry size
----------------------------------------------------------------
However, the current implementation of OOM which has been in use for
almost a decade scores based on 1/10 % of memory. This methodology scales
well as memory sizes increase. If you limit the objects you examine to
those using 0.1% of memory you still may get a large number of objects
but avoid printing those using a relatively small amount of memory.

Further options that allow limiting output based on object size
can have the minimum size set to zero. In this case objects
that use even a small amount of memory will be printed.

Use of debugfs to allow dynamic controls
----------------------------------------
By providing a debugfs interface that allows options to be configured,
enabled and where appropriate to set a minimum size for selecting
entries to print, the output produced when an OOM event occurs can be
dynamically adjusted to produce as little or as much detail as needed
for a given system.

OOM debug options can be added to the base code as needed.

Currently we have the following OOM debug options defined:

* System State Summary
  --------------------
  One line of output that includes:
  - Uptime (days, hour, minutes, seconds)
  - Number CPUs
  - Machine Type
  - Node name
  - Domain name
  - Kernel Release
  - Kernel Version

  Example output when configured and enabled:

Jul 27 10:56:46 yoursystem kernel: System Uptime:0 days 00:17:27 CPUs:4 Machine:x86_64 Node:yoursystem Domain:localdomain Kernel Release:5.3.0-rc2+ Version: #49 SMP Mon Jul 27 10:35:32 PDT 2019

* Tasks Summary
  -------------
  One line of output that includes:
  - Number of Threads
  - Number of processes
  - Forks since boot
  - Processes that are runnable
  - Processes that are in iowait

  Example output when configured and enabled:

Jul 22 15:20:57 yoursystem kernel: Threads:530 Processes:279 forks_since_boot:2786 procs_runable:2 procs_iowait:0

* ARP Table and/or Neighbour Discovery Table Summary
  --------------------------------------------------
  One line of output each for ARP and ND that includes:
  - Table name
  - Table size (max # entries)
  - Key Length
  - Entry Size
  - Number of Entries
  - Last Flush (in seconds)
  - hash grows
  - entry allocations
  - entry destroys
  - Number lookups
  - Number of lookup hits
  - Resolution failures
  - Garbage Collection Forced Runs
  - Table Full
  - Proxy Queue Length

  Example output when configured and enabled (for both):

... kernel: neighbour: Table: arp_tbl size:   256 keyLen:  4 entrySize: 360 entries:     9 lastFlush:  1721s hGrows:     1 allocs:     9 destroys:     0 lookups:   204 hits:   199 resFailed:    38 gcRuns/Forced: 111 /  0 tblFull:  0 proxyQlen:  0

... kernel: neighbour: Table:  nd_tbl size:   128 keyLen: 16 entrySize: 368 entries:     6 lastFlush:  1720s hGrows:     0 allocs:     7 destroys:     1 lookups:     0 hits:     0 resFailed:     0 gcRuns/Forced: 110 /  0 tblFull:  0 proxyQlen:  0

* Add Select Slabs Print
  ----------------------
  Allow select slab entries (based on a minimum size) to be printed.
  Minimum size is specified as a percentage of the total RAM memory
  in tenths of a percent, consistent with existing OOM process scoring.
  Valid values are specified from 0 to 1000 where 0 prints all slab
  entries (all slabs that have at least one slab object in use) up
  to 1000 which would require a slab to use 100% of memory which can't
  happen so in that case only summary information is printed.

  The first line of output is the standard Linux output header for
  OOM printed Slab entries. This header looks like this:

Aug  6 09:37:21 egc103 yourserver: Unreclaimable slab info:

  The output is existing slab entry memory usage limited such that only
  entries equal to or larger than the minimum size are printed.
  Empty slabs (no slab entries in slabs in use) are never printed.

  Additional output consists of summary information that is printed
  at the end of the output. This summary information includes:
  - # entries examined
  - # entries selected and printed
  - minimum entry size for selection
  - Slabs total size (kB)
  - Slabs reclaimable size (kB)
  - Slabs unreclaimable size (kB)

  Example Summary output when configured and enabled:

Jul 23 23:26:34 yoursystem kernel: Summary: Slab entries examined: 123 printed: 83 minsize: 0kB

Jul 23 23:26:34 yoursystem kernel: Slabs Total: 151212kB Reclaim: 50632kB Unreclaim: 100580kB

* Add Select Vmalloc allocations Print
  ------------------------------------
  Allow select vmalloc entries (based on a minimum size) to be printed.
  Minimum size is specified as a percentage of the total RAM memory
  in tenths of a percent, consistent with existing OOM process scoring.
  Valid values are specified from 0 to 1000 where 0 prints all vmalloc
  entries (all vmalloc allocations that have at least one page in use) up
  to 1000 which would require a vmalloc to use 100% of memory which can't
  happen so in that case only summary information is printed.

  The first line of output is a new Vmalloc output header for
  OOM printed Vmalloc entries. This header looks like this:

Aug 19 19:27:01 yourserver kernel: Vmalloc Info:

  The output is vmalloc entry information output limited such that only
  entries equal to or larger than the minimum size are printed.
  Unused vmallocs (no pages assigned to the vmalloc) are never printed.
  The vmalloc entry information includes:
  - Size (in bytes)
  - pages (Number pages in use)
  - Caller Information to identify the request

  A sample vmalloc entry output looks like this:

Jul 22 20:16:09 yoursystem kernel: Vmalloc size=2625536 pages=640 caller=__do_sys_swapon+0x78e/0x113

  Additional output consists of summary information that is printed
  at the end of the output. This summary information includes:
  - Number of Vmalloc entries examined
  - Number of Vmalloc entries printed
  - minimum entry size for selection

  A sample Vmalloc Summary output looks like this:

Aug 19 19:27:01 coronado kernel: Summary: Vmalloc entries examined: 1070 printed: 989 minsize: 0kB

* Add Select Process Entries Print
  --------------------------------
  Allow select process entries (based on a minimum size) to be printed.
  Minimum size is specified as a percentage totalpages (RAM + swap)
  in tenths of a percent, consistent with existing OOM process scoring.
  Note: user process memory can be swapped out when swap space present
  so that is why swap space and ram memory comprise the totalpages
  used to calculate the percentage of memory a process is using.
  Valid values are specified from 0 to 1000 where 0 prints all user
  processes (that have valid mm sections and aren't exiting) up to
  1000 which would require a user process to use 100% of memory which
  can't happen so in that case only summary information is printed.

  The first line of output is the standard Linux output headers for
  OOM printed User Processes. This header looks like this:

Aug 19 19:27:01 yourserver kernel: Tasks state (memory values in pages):
Aug 19 19:27:01 yourserver kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name

  The output is existing per user process data limited such that only
  entries equal to or larger than the minimum size are printed.

Jul 21 20:07:48 yourserver kernel: [    579]     0   579     7942     1010          90112        0         -1000 systemd-udevd

  Additional output consists of summary information that is printed
  at the end of the output. This summary information includes:

Aug 19 19:27:01 yourserver kernel: Summary: OOM Tasks considered:277 printed:143 minimum size:0kB totalpages:32791608kB

* Add Slab Select Always Print Enable
  -----------------------------------
  This option will enable slab entries to be printed even when slab
  memory usage does not exceed the standard Linux user memory usage
  print trigger. The Standard OOM event Slab entry print trigger is
  that slab memory usage exceeds user memory usage. This covers cases
  where the Kernel or Kernel drivers are driving slab memory usage up
  causing it to be excessive. However, OOM Events are often caused by
  user processes causing too much memory usage. In some cases where
  the user memory usage is higher the amount of slab memory consumed
  can still be an important factor in determining what caused the OOM
  event. In such cases it would be useful to have slab memory usage
  for any slab entries using a significant amount of memory.

  No changes to output format occurs, enabling the option simply
  causes what ever slabs are print eligible (from Select Slabs
  option, which this option depends on) get printed on any OOM
  event regardless of whether the memory usage by Slabs exceeds
  user memory usage or not.

* Add Enhanced Slab Print Information
  -----------------------------------
  For any slab entries that are print eligible (from Select Slabs
  option, which this option depends on) print some additional
  details about the slab that can be useful to root causing
  OOM events.

  Output information for each enhanced slab entry includes:
  - Used space (KiB)
  - Total space (KiB)
  - Active objects
  - Total Objects
  - Object size
  - Aligned object size
  - Object per Slab
  - Pages per Slab
  - Active Slabs
  - Total Slabs
  - Slab name

  The header for enhanced slab entries is revised and looks like this:

Aug 19 19:27:01 coronado kernel:   UsedKiB   TotalKiB  ActiveObj   TotalObj   ObjSize AlignSize Objs/Slab Pgs/Slab ActiveSlab  TotalSlab Slab_Name

  Each enhanced slab entry is similar to the following output format:

Aug 19 19:27:01 coronado kernel:      9016       9016     384710     384710        24        24       170        1       2263       2263 avtab_node


* Add Enhanced Process Print Information
  --------------------------------------
  Add OOM Debug code that prints additional detailed information about
  users processes that were considered for OOM killing for any print
  selected processes. The information is displayed for each user process
  that OOM prints in the output.

  This supplemental per user process information is very helpful for
  determing how process memory is used to allow OOM event root cause
  identifcation that might not otherwise be possible.

  Output information for enhanced user process entrys printed includes:
  - pid
  - parent pid
  - ruid
  - euid
  - tgid
  - Process State (S)
  - utime in seconds
  - stime in seconds
  - oom_score_adjust
  - task comm value (name of process)
  - Vmem KiB
  - MaxRss KiB
  - CurRss KiB
  - Pte KiB
  - Swap KiB
  - Sock KiB
  - Lib KiB
  - Text KiB
  - Heap KiB
  - Stack KiB
  - File KiB
  - Shmem KiB
  - Read Pages
  - Fault Pages
  - Lock KiB
  - Pinned KiB

  The headers for Processes changes to match the data being printed:

Aug 19 19:27:01 yourserver kernel: Tasks state (memory values in KiB):

...: [  pid  ]    ppid    ruid    euid    tgid S  utimeSec  stimeSec   VmemKiB MaxRssKiB CurRssKiB    PteKiB   SwapKiB   SockKiB     LibKiB   TextKiB   HeapKiB  StackKiB   FileKiB  ShmemKiB     ReadPgs    FaultPgs   LockKiB PinnedKiB Adjust name

  A few entries that print formatted to match the second header:

...: [    570]       1       0       0     570 S     0.530     0.105     31632     12064      3864        88         0       416       9500       208      3608       132        36         0          60       41615         0         0  -1000 systemd-udevd
...: [    759]       1       0       0     759 S     1.264     0.545     17196      6072       788        72         0       624       8912        32       596       132         0         0           0           0         0         0      0 rngd
...: [   1626]    1553   10383   10383    1626 S     9.417     2.355   3347904    336316    231672       924         0       416      56452        16    170656       276      2116    150756           4        2309         0         0      0 gnome-shell

Configuring Patches:
-------------------
OOM Debug and any options you want to use must first be configured so
the code is included in your kernel. This requires selecting kernel
config file options. You will find config options to select under:

Kernel hacking ---> Memory Debugging --->

[*] Debug OOM
    [*] Debug OOM System State
    [*] Debug OOM System Tasks Summary
    [*] Debug OOM ARP Table
    [*] Debug OOM ND Table
    [*] Debug OOM Select Slabs Print
       [*] Debug OOM Slabs Select Always Print Enable
       [*] Debug OOM Enhanced Slab Print
    [*] Debug OOM Select Vmallocs Print
    [*] Debug OOM Select Process Print
       [*] Debug OOM Enhanced Process Print

The heirarchy shown also displays the dependencies between OOM Debug for
these options. Everything depends on Debug OOM as that is where the base
code that all options require is located. Process has an Enhanced output
but requires Select Process to be enabled so you can limit the output
since you're asking for more details. The same is true with Slabs the
Enhanced output requires Select Slabs and so does Slabs Select Always
Print, to ensure you can limit your output if you need to.

Dyanmic enable/disable and setting entry minsize for Options
------------------------------------------------------------
As mentioned all options can be dynamically disabled and re-enabled.
The Select Options also allow setting minimum entry size to limit
entry printing based on the amount of memory they use, using the
OOM 0% to 100% in 1/10 % increments (1-1000). This is impelemented in
debugfs. Entries for OOM Debug are defined in the /sys/kernel/debug/oom
directory.

Arbitrary default values have been selected. The default is to enable
configured options and to set minimum entry size to 10 which is 1% of
the memory (or memory plus swap for processes). The choice was to
make sure by default you don't get a lot of data just for enabling an
option. Here is what the current defaults are set to for all the
OOM Debug options we currently have defined:

[root@yourserver ~]# grep "" /sys/kernel/debug/oom/*
/sys/kernel/debug/oom/arp_table_summary_enabled:Y
/sys/kernel/debug/oom/nd_table_summary_enabled:Y
/sys/kernel/debug/oom/process_enhanced_print_enabled:Y
/sys/kernel/debug/oom/process_select_print_enabled:Y
/sys/kernel/debug/oom/process_select_print_tenthpercent:10
/sys/kernel/debug/oom/slab_enhanced_print_enabled:Y
/sys/kernel/debug/oom/slab_select_always_print_enabled:Y
/sys/kernel/debug/oom/slab_select_print_enabled:Y
/sys/kernel/debug/oom/slab_select_print_tenthpercent:10
/sys/kernel/debug/oom/system_state_summary_enabled:Y
/sys/kernel/debug/oom/tasks_summary_enabled:Y
/sys/kernel/debug/oom/vmalloc_select_print_enabled:Y
/sys/kernel/debug/oom/vmalloc_select_print_tenthpercent:10

You can disable or re-enable options in the appropriate enable file
or adjust the minimum size value in the appropriate tenthpercent file
as needed.

---------------------------------------------------------------------

Edward Chron (10):
  mm/oom_debug: Add Debug base code
  mm/oom_debug: Add System State Summary
  mm/oom_debug: Add Tasks Summary
  mm/oom_debug: Add ARP and ND Table Summary usage
  mm/oom_debug: Add Select Slabs Print
  mm/oom_debug: Add Select Vmalloc Entries Print
  mm/oom_debug: Add Select Process Entries Print
  mm/oom_debug: Add Slab Select Always Print Enable
  mm/oom_debug: Add Enhanced Slab Print Information
  mm/oom_debug: Add Enhanced Process Print Information

 include/linux/oom.h     |   1 +
 include/linux/vmalloc.h |  12 +
 include/net/neighbour.h |  12 +
 mm/Kconfig.debug        | 228 +++++++++++++
 mm/Makefile             |   1 +
 mm/oom_kill.c           |  83 ++++-
 mm/oom_kill_debug.c     | 736 ++++++++++++++++++++++++++++++++++++++++
 mm/oom_kill_debug.h     |  58 ++++
 mm/slab.h               |   4 +
 mm/slab_common.c        |  94 +++++
 mm/vmalloc.c            |  43 +++
 net/core/neighbour.c    |  78 +++++
 12 files changed, 1339 insertions(+), 11 deletions(-)
 create mode 100644 mm/oom_kill_debug.c
 create mode 100644 mm/oom_kill_debug.h

-- 
2.20.1