Re: [PACTH v2 0/3] Implement /proc/<pid>/totmaps

[Michal Hocko]

[Sorry for a late reply, I was busy with other stuff]

On Mon 22-08-16 15:44:53, Sonny Rao wrote:
> On Mon, Aug 22, 2016 at 12:54 AM, Michal Hocko <mhocko@xxxxxxxxxx> wrote:
[...]
> But what about the private_clean and private_dirty?  Surely
> those are more generally useful for calculating a lower bound on
> process memory usage without additional knowledge?

I guess private_clean can be used as a reasonable estimate.
private_dirty less so because it may refer to e.g. tmpfs which is not
mapped by other process and so no memory would be freed after unmap
without removing the file.

> At the end of the day all of these metrics are approximations, and it
> comes down to how far off the various approximations are and what
> trade offs we are willing to make.
> RSS is the cheapest but the most coarse.

I agree on this part definitely. I also understand that what we provide
currently is quite confusing and not really helpful. But I am afraid
that the accounting is far from trivial to make right for all the
possible cases.

> PSS (with the correct context) and Private data plus swap are much
> better but also more expensive due to the PT walk.

Maybe we can be more clever and do some form of caching. I haven't
thought that through to see how hard that could be. I mean we could
cache some data per mm_struct and invalidate them only after the current
value would get too much out of sync.

> As far as I know, to get anything but RSS we have to go through smaps
> or use memcg.  Swap seems to be available in /proc/<pid>/status.
> 
> I looked at the "shared" value in /proc/<pid>/statm but it doesn't
> seem to correlate well with the shared value in smaps -- not sure why?

task_statm() does only approximate to get_mm_counter(mm, MM_FILEPAGES) +
get_mm_counter(mm, MM_SHMEMPAGES) so all the pages accounted to the mm.
If they are not shared by anybody else they would be considered private
by smaps.

> It might be useful to show the magnitude of difference of using RSS vs
> PSS/Private in the case of the Chrome renderer processes.  On the
> system I was looking at there were about 40 of these processes, but I
> picked a few to give an idea:
> 
> localhost ~ # cat /proc/21550/totmaps
> Rss:               98972 kB
> Pss:               54717 kB
> Shared_Clean:      19020 kB
> Shared_Dirty:      26352 kB
> Private_Clean:         0 kB
> Private_Dirty:     53600 kB
> Referenced:        92184 kB
> Anonymous:         46524 kB
> AnonHugePages:     24576 kB
> Swap:              13148 kB
> 
> 
> RSS is 80% higher than PSS and 84% higher than private data
> 
> localhost ~ # cat /proc/21470/totmaps
> Rss:              118420 kB
> Pss:               70938 kB
> Shared_Clean:      22212 kB
> Shared_Dirty:      26520 kB
> Private_Clean:         0 kB
> Private_Dirty:     69688 kB
> Referenced:       111500 kB
> Anonymous:         79928 kB
> AnonHugePages:     24576 kB
> Swap:              12964 kB
> 
> RSS is 66% higher than RSS and 69% higher than private data
> 
> localhost ~ # cat /proc/21435/totmaps
> Rss:               97156 kB
> Pss:               50044 kB
> Shared_Clean:      21920 kB
> Shared_Dirty:      26400 kB
> Private_Clean:         0 kB
> Private_Dirty:     48836 kB
> Referenced:        90012 kB
> Anonymous:         75228 kB
> AnonHugePages:     24576 kB
> Swap:              13064 kB
> 
> RSS is 94% higher than PSS and 98% higher than private data.
> 
> It looks like there's a set of about 40MB of shared pages which cause
> the difference in this case.
> Swap was roughly even on these but I don't think it's always going to be true.

OK, I see that those processes differ in the way how they are using
memory but I am not really sure what kind of conclusion you can draw
from that.
-- 
Michal Hocko
SUSE Labs