Re: [PATCH v2] docs: proc.rst: meminfo: briefly describe gaps in memory accounting

[Michal Hocko]

On Tue 20-04-21 14:24:30, Matthew Wilcox wrote:
> On Tue, Apr 20, 2021 at 03:13:54PM +0300, Mike Rapoport wrote:
> > Add a paragraph that explains that it may happen that the counters in
> > /proc/meminfo do not add up to the overall memory usage.
> 
> ... that is, the sum may be lower because memory is allocated for other
> purposes that is not reported here, right?

yes. Many direct page allocator users are not accounted in any of the
existing counters.

> Is it ever possible for it to be higher?  Maybe due to a race when
> sampling the counters?

Yes likely possible. You will never get an atomic snapshot of all
counters.

> >  Provides information about distribution and utilization of memory.  This
> > -varies by architecture and compile options.  The following is from a
> > -16GB PIII, which has highmem enabled.  You may not have all of these fields.
> > +varies by architecture and compile options. Please note that it may happen
> > +that the memory accounted here does not add up to the overall memory usage
> > +and the difference for some workloads can be substantial. In many cases there
> > +are other means to find out additional memory using subsystem specific
> > +interfaces, for instance /proc/net/sockstat for TCP memory allocations.
> 
> How about just:
> 
> +varies by architecture and compile options.  The memory reported here
> +may not add up to the overall memory usage and the difference for some
> +workloads can be substantial. [...]
> 
> But I'd like to be a bit more explicit about the reason, hence my question
> above to be sure I understand.
> 
> 
> It's also not entirely clear which of the fields in meminfo can be
> usefully summed.  VmallocTotal is larger than MemTotal, for example.

Yes. Many/Most counters cannot be simply sumed up. A trivial example would be
Active/Inactive is a sum of both anona and file. Mlocked will be
accounted in LRU pages and Unevictable. MemAvailable is not really a
counter... 

Usual memory consumption is usually something like LRU pages + Slab
memory + kernel stack + vmalloc used + pcp.

> But I know that KernelStack is allocated through vmalloc these days,
> and I don't know whether VmallocUsed includes KernelStack or whether I
> can sum them.  Similarly, is Mlocked a subset of Unevictable?
> 
> There is some attempt at explaining how these numbers fit together, but
> it's outdated, and doesn't include Mlocked, Unevictable or KernelStack

Agreed there is a lot of tribal knowledge or even misconceptions flying
around and it will take much more work to put everything into shape.
This is only one tiny step forward.
-- 
Michal Hocko
SUSE Labs