Re: [PATCH v7 5/6] cgroup/cpuset: Update description of cpuset.cpus.partition in cgroup-v2.rst

[Michal Koutný]

On Wed, Oct 06, 2021 at 02:21:03PM -0400, Waiman Long <llong@xxxxxxxxxx> wrote:
> Sorry for not following up with this patchset sooner as I was busy on other
> tasks.

Thanks for continuing with this.

> 	1) The "cpuset.cpus" is not empty and the list of CPUs are
> 	   exclusive, i.e. they are not shared by any of its siblings.
> 	2) The parent cgroup is a partition root.
> 	3) The "cpuset.cpus" is a subset of the union of parent's
> 	   "cpuset.cpus.effective" and offlined CPUs in parent's
> 	   "cpuset.cpus".
> 	4) There is no child cgroups with cpuset enabled.  This avoids
> 	   cpu migrations of multiple cgroups simultaneously which can
> 	   be problematic.
> 
>         A partition, when enabled, can be in an invalid state. An example
>         is when its parent is also an invalid partition.

You say:
"it can only be enabled in a cgroup if all the following conditions are met.",
"2) The parent cgroup is a partition root."

and then the example:
"A partition, when enabled, can be in an invalid state. An example is
when its parent is also an invalid partition."

But the first two statements imply you can't have enabled the partition
in such a case.

I think there is still mixup of partition validity conditions and
transition conditions, yours would roughly divide into (not precisely,
just to share my understanding):

Validity conditions
 	1) The "cpuset.cpus" is not empty and the list of CPUs are
 	   exclusive, i.e. they are not shared by any of its siblings.
 	2) The parent cgroup is a partition root.

Transition conditions:
 	3) The "cpuset.cpus" is a subset of the union of parent's
 	   "cpuset.cpus.effective" and offlined CPUs in parent's
 	   "cpuset.cpus".
 	4) There is no child cgroups with cpuset enabled.  This avoids
 	   cpu migrations of multiple cgroups simultaneously which can
 	   be problematic.

(I've put no. 3 into transition conditions because _after_ the
transition parent's cpuset.cpus.effective are subtracted the new root's
cpuset.cpus but I'd like to have something similar as a validity
condition but I haven't come up with that yet.)

I consider the following situation:

r		// all cpus 0-7
`- part1	cpus=0-3	root >partition
   ` subpart1	cpus=0-1	root >partition
   ` subpart2	cpus=2-3	root >partition
`- other	cpus=4-7	// member by default

Both subpart1 and subpart2 are valid partition roots.
Look at actions listed below (as alternatives, not a sequence):

a) hotplug offlines cpu 3
  - would part1 still be considered a valid root? 
    - perhaps not
  - would subpart1 still be considered a valid root? 
    - it could be, but its parent is invalid so no?
  - would subpart2 still be considered a valid root? 
    - perhaps not
    
b) administrative change writes 0-2 into part1 cpus
  - would part1 still be considered a valid root? 
    - yes
  - would subpart1 still be considered a valid root? 
    - yes
  - would subpart2 still be considered a valid root? 
    - perhaps not

c) administrative change writes 3-7 into `other` cpus
  - should this fail or invalidate a root partition part1?
    - perhaps fail since the same "owner" manages all siblings and
      should reduce part1 first

The answers above are just my "natural" responses, the ideal may be
different. The issue I want to illustrate is that if all the conditions
are formed as transition conditions only, they can't be used to reason
about hotplug or config changes (except for cpuset.cpus.partitions
writes).

What would help me with the understanding -- the invalid root partition is defined as
1) such a cgroup where no cpus are granted from the top (and thus has to fall back to ancestors)
or
2) such a cgroup where cpus requested in cpuset.cpus can't be fulfilled (i.e. any missing invalidates)?

Furthermore, another example (motivated by the patch 4/6)

r		// all cpus 0-7
`- part1	cpus=0-4	root >partition
   ` subpart1	cpus=0-1	root >partition
   ` subpart2	cpus=2-3	root >partition
   ` task
`- other	cpus=5-7	// member by default

It's a valid and achievable state (even on v2 since cpuset is a threaded
controller). 

a) cpu 4 is offlined
  - this should invalidate part1 (and propagate invalidation into
    subpart1 and subpart2).
b) administrative write 0-3 into part1 cpus
  - should this invalidate part1 or be rejected?


In conclusion, it'd be good to have validity conditions separate from
transition conditions (since hotplug transition can't be rejected) and
perhaps treat administrative changes from an ancestor equally as a
hotplug.

Thanks,
Michal