Re: [PATCH v3 02/11] memcg: document cgroup dirty memory interfaces

[KAMEZAWA Hiroyuki]

On Tue, 19 Oct 2010 14:00:58 -0700
Greg Thelen <gthelen@xxxxxxxxxx> wrote:

> Daisuke Nishimura <nishimura@xxxxxxxxxxxxxxxxx> writes:
> 
> > On Mon, 18 Oct 2010 17:39:35 -0700
> > Greg Thelen <gthelen@xxxxxxxxxx> wrote:
> >
> >> Document cgroup dirty memory interfaces and statistics.
> >> 
> >> Signed-off-by: Andrea Righi <arighi@xxxxxxxxxxx>
> >> Signed-off-by: Greg Thelen <gthelen@xxxxxxxxxx>
> >> ---
> >> 
> >> Changelog since v1:
> >> - Renamed "nfs"/"total_nfs" to "nfs_unstable"/"total_nfs_unstable" in per cgroup
> >>   memory.stat to match /proc/meminfo.
> >> 
> >> - Allow [kKmMgG] suffixes for newly created dirty limit value cgroupfs files.
> >> 
> >> - Describe a situation where a cgroup can exceed its dirty limit.
> >> 
> >>  Documentation/cgroups/memory.txt |   60 ++++++++++++++++++++++++++++++++++++++
> >>  1 files changed, 60 insertions(+), 0 deletions(-)
> >> 
> >> diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt
> >> index 7781857..02bbd6f 100644
> >> --- a/Documentation/cgroups/memory.txt
> >> +++ b/Documentation/cgroups/memory.txt
> >> @@ -385,6 +385,10 @@ mapped_file	- # of bytes of mapped file (includes tmpfs/shmem)
> >>  pgpgin		- # of pages paged in (equivalent to # of charging events).
> >>  pgpgout		- # of pages paged out (equivalent to # of uncharging events).
> >>  swap		- # of bytes of swap usage
> >> +dirty		- # of bytes that are waiting to get written back to the disk.
> >> +writeback	- # of bytes that are actively being written back to the disk.
> >> +nfs_unstable	- # of bytes sent to the NFS server, but not yet committed to
> >> +		the actual storage.
> >>  inactive_anon	- # of bytes of anonymous memory and swap cache memory on
> >>  		LRU list.
> >>  active_anon	- # of bytes of anonymous and swap cache memory on active
> >
> > Shouldn't we add description of "total_diryt/writeback/nfs_unstable" too ?
> > Seeing [5/11], it will be showed in memory.stat.
> 
> Good catch.  See patch (below).
> 
> >> @@ -453,6 +457,62 @@ memory under it will be reclaimed.
> >>  You can reset failcnt by writing 0 to failcnt file.
> >>  # echo 0 > .../memory.failcnt
> >>  
> >> +5.5 dirty memory
> >> +
> >> +Control the maximum amount of dirty pages a cgroup can have at any given time.
> >> +
> >> +Limiting dirty memory is like fixing the max amount of dirty (hard to reclaim)
> >> +page cache used by a cgroup.  So, in case of multiple cgroup writers, they will
> >> +not be able to consume more than their designated share of dirty pages and will
> >> +be forced to perform write-out if they cross that limit.
> >> +
> >> +The interface is equivalent to the procfs interface: /proc/sys/vm/dirty_*.  It
> >> +is possible to configure a limit to trigger both a direct writeback or a
> >> +background writeback performed by per-bdi flusher threads.  The root cgroup
> >> +memory.dirty_* control files are read-only and match the contents of
> >> +the /proc/sys/vm/dirty_* files.
> >> +
> >> +Per-cgroup dirty limits can be set using the following files in the cgroupfs:
> >> +
> >> +- memory.dirty_ratio: the amount of dirty memory (expressed as a percentage of
> >> +  cgroup memory) at which a process generating dirty pages will itself start
> >> +  writing out dirty data.
> >> +
> >> +- memory.dirty_limit_in_bytes: the amount of dirty memory (expressed in bytes)
> >> +  in the cgroup at which a process generating dirty pages will start itself
> >> +  writing out dirty data.  Suffix (k, K, m, M, g, or G) can be used to indicate
> >> +  that value is kilo, mega or gigabytes.
> >> +
> >> +  Note: memory.dirty_limit_in_bytes is the counterpart of memory.dirty_ratio.
> >> +  Only one of them may be specified at a time.  When one is written it is
> >> +  immediately taken into account to evaluate the dirty memory limits and the
> >> +  other appears as 0 when read.
> >> +
> >> +- memory.dirty_background_ratio: the amount of dirty memory of the cgroup
> >> +  (expressed as a percentage of cgroup memory) at which background writeback
> >> +  kernel threads will start writing out dirty data.
> >> +
> >> +- memory.dirty_background_limit_in_bytes: the amount of dirty memory (expressed
> >> +  in bytes) in the cgroup at which background writeback kernel threads will
> >> +  start writing out dirty data.  Suffix (k, K, m, M, g, or G) can be used to
> >> +  indicate that value is kilo, mega or gigabytes.
> >> +
> >> +  Note: memory.dirty_background_limit_in_bytes is the counterpart of
> >> +  memory.dirty_background_ratio.  Only one of them may be specified at a time.
> >> +  When one is written it is immediately taken into account to evaluate the dirty
> >> +  memory limits and the other appears as 0 when read.
> >> +
> >> +A cgroup may contain more dirty memory than its dirty limit.  This is possible
> >> +because of the principle that the first cgroup to touch a page is charged for
> >> +it.  Subsequent page counting events (dirty, writeback, nfs_unstable) are also
> >> +counted to the originally charged cgroup.
> >> +
> >> +Example: If page is allocated by a cgroup A task, then the page is charged to
> >> +cgroup A.  If the page is later dirtied by a task in cgroup B, then the cgroup A
> >> +dirty count will be incremented.  If cgroup A is over its dirty limit but cgroup
> >> +B is not, then dirtying a cgroup A page from a cgroup B task may push cgroup A
> >> +over its dirty limit without throttling the dirtying cgroup B task.
> >> +
> >>  6. Hierarchy support
> >>  
> >>  The memory controller supports a deep hierarchy and hierarchical accounting.
> >> -- 
> >> 1.7.1
> >> 
> > Can you clarify whether we can limit the "total" dirty pages under hierarchy
> > in use_hierarchy==1 case ?
> > If we can, I think it would be better to note it in this documentation.
> >
> >
> > Thanks,
> > Daisuke Nishimura.
> 
> Here is a second version of this -v3 doc patch:
> 
> Author: Greg Thelen <gthelen@xxxxxxxxxx>
> Date:   Sat Apr 10 15:34:28 2010 -0700
> 
>     memcg: document cgroup dirty memory interfaces
>     
>     Document cgroup dirty memory interfaces and statistics.
>     
>     Signed-off-by: Andrea Righi <arighi@xxxxxxxxxxx>
>     Signed-off-by: Greg Thelen <gthelen@xxxxxxxxxx>
> 

nitpicks. and again, why you always drop Acks ?

> diff --git a/Documentation/cgroups/memory.txt b/Documentation/cgroups/memory.txt
> index 7781857..8bf6d3b 100644
> --- a/Documentation/cgroups/memory.txt
> +++ b/Documentation/cgroups/memory.txt
> @@ -385,6 +385,10 @@ mapped_file	- # of bytes of mapped file (includes tmpfs/shmem)
>  pgpgin		- # of pages paged in (equivalent to # of charging events).
>  pgpgout		- # of pages paged out (equivalent to # of uncharging events).
>  swap		- # of bytes of swap usage
> +dirty		- # of bytes that are waiting to get written back to the disk.

extra tab ?

> +writeback	- # of bytes that are actively being written back to the disk.
> +nfs_unstable	- # of bytes sent to the NFS server, but not yet committed to
> +		the actual storage.
>  inactive_anon	- # of bytes of anonymous memory and swap cache memory on
>  		LRU list.
>  active_anon	- # of bytes of anonymous and swap cache memory on active
> @@ -406,6 +410,9 @@ total_mapped_file	- sum of all children's "cache"
>  total_pgpgin		- sum of all children's "pgpgin"
>  total_pgpgout		- sum of all children's "pgpgout"
>  total_swap		- sum of all children's "swap"
> +total_dirty		- sum of all children's "dirty"
> +total_writeback		- sum of all children's "writeback"

here, too.

> +total_nfs_unstable	- sum of all children's "nfs_unstable"
>  total_inactive_anon	- sum of all children's "inactive_anon"
>  total_active_anon	- sum of all children's "active_anon"
>  total_inactive_file	- sum of all children's "inactive_file"
> @@ -453,6 +460,71 @@ memory under it will be reclaimed.
>  You can reset failcnt by writing 0 to failcnt file.
>  # echo 0 > .../memory.failcnt
>  
> +5.5 dirty memory
> +
> +Control the maximum amount of dirty pages a cgroup can have at any given time.
> +
> +Limiting dirty memory is like fixing the max amount of dirty (hard to reclaim)
> +page cache used by a cgroup.  So, in case of multiple cgroup writers, they will
> +not be able to consume more than their designated share of dirty pages and will
> +be forced to perform write-out if they cross that limit.
> +
> +The interface is equivalent to the procfs interface: /proc/sys/vm/dirty_*.  It
> +is possible to configure a limit to trigger both a direct writeback or a
> +background writeback performed by per-bdi flusher threads.  The root cgroup
> +memory.dirty_* control files are read-only and match the contents of
> +the /proc/sys/vm/dirty_* files.
> +
> +Per-cgroup dirty limits can be set using the following files in the cgroupfs:
> +
> +- memory.dirty_ratio: the amount of dirty memory (expressed as a percentage of
> +  cgroup memory) at which a process generating dirty pages will itself start
> +  writing out dirty data.
> +
> +- memory.dirty_limit_in_bytes: the amount of dirty memory (expressed in bytes)
> +  in the cgroup at which a process generating dirty pages will start itself
> +  writing out dirty data.  Suffix (k, K, m, M, g, or G) can be used to indicate
> +  that value is kilo, mega or gigabytes.
> +
> +  Note: memory.dirty_limit_in_bytes is the counterpart of memory.dirty_ratio.
> +  Only one of them may be specified at a time.  When one is written it is
> +  immediately taken into account to evaluate the dirty memory limits and the
> +  other appears as 0 when read.
> +
> +- memory.dirty_background_ratio: the amount of dirty memory of the cgroup
> +  (expressed as a percentage of cgroup memory) at which background writeback
> +  kernel threads will start writing out dirty data.
> +
> +- memory.dirty_background_limit_in_bytes: the amount of dirty memory (expressed
> +  in bytes) in the cgroup at which background writeback kernel threads will
> +  start writing out dirty data.  Suffix (k, K, m, M, g, or G) can be used to
> +  indicate that value is kilo, mega or gigabytes.
> +
> +  Note: memory.dirty_background_limit_in_bytes is the counterpart of
> +  memory.dirty_background_ratio.  Only one of them may be specified at a time.
> +  When one is written it is immediately taken into account to evaluate the dirty
> +  memory limits and the other appears as 0 when read.
> +
> +A cgroup may contain more dirty memory than its dirty limit.  This is possible
> +because of the principle that the first cgroup to touch a page is charged for
> +it.  Subsequent page counting events (dirty, writeback, nfs_unstable) are also
> +counted to the originally charged cgroup.
> +
> +Example: If page is allocated by a cgroup A task, then the page is charged to
> +cgroup A.  If the page is later dirtied by a task in cgroup B, then the cgroup A
> +dirty count will be incremented.  If cgroup A is over its dirty limit but cgroup
> +B is not, then dirtying a cgroup A page from a cgroup B task may push cgroup A
> +over its dirty limit without throttling the dirtying cgroup B task.
> +
> +When use_hierarchy=0, each cgroup has independent dirty memory usage and limits.
> +
> +When use_hierarchy=1, a parent cgroup increasing its dirty memory usage will
> +compare its total_dirty memory (which includes sum of all child cgroup dirty
> +memory) to its dirty limits.  This keeps a parent from explicitly exceeding its
> +dirty limits.  However, a child cgroup can increase its dirty usage without
> +considering the parent's dirty limits.  Thus the parent's total_dirty can exceed
> +the parent's dirty limits as a child dirties pages.

Hmm. in short, dirty_ratio in use_hierarchy=1 doesn't work as an user expects.
Is this a spec. or a current implementation ?

I think as following.
 - add a limitation as "At setting chidlren's dirty_ratio, it must be below parent's.
   If it exceeds parent's dirty_ratio, EINVAL is returned."

Could you modify setting memory.dirty_ratio code ?
Then, parent's dirty_ratio will never exceeds its own. (If I understand correctly.)

"memory.dirty_limit_in_bytes" will be a bit more complecated, but I think you can.


Thanks,
-Kame

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/