Re: [RFC PATCH 0/9 v2] mm/memcontrol: Make memory cgroup limits tier-aware

[Joshua Hahn]

Sorry, re-sending because linux-mm didn't get added to the reply when I
sent it out.

> Sorry for the late reply :) Yes, hopefully talk to you in person next time!

Hello David, no worries! Thank you for taking the time to consider this
series again.

> >> Assume you have a bigger hierarchy (HBP, DRAM, CXL), or assume you have multiple
> >> NUMA nodes with a hierarchy each.
> >>
> >> Your proposal doesn't really seem to be very versatile, or am I wrong?
> > 
> > Let me address these comments separately!
> > 
> > First, for the multi-numa-per-tier case, I think this is already pretty well
> > handled by my series. Once we realize that a memcg is consuming too much memory
> > from a tier, we trigger reclaim from that memcg via
> > try_to_free_mem_cgroup_pages,
> > which as far as I can tell already handles the multi-numa per memcg case.
> > Other than restricting the scan_control's nodemask to target the nodes
> > from that tier, I don't think there's anything else to be done.
> 
> I am not really sure about the "toptier" notion, in particular if your memory
> hierarchy is not static.
> 
> Imagine that you have only DRAM in your system. Then you hotplug some slow CXL
> memory. The you hotplug some HBM CXL memory.
> 
> The meanting of "toptier" changes, which is rather confusing to me.

While I do agree with the idea that tiers in general are not stable, I think
that for the most part the "toptier" node is generally stable through hotplugging
nodes. In your example above, the DRAM will always be the toptier node, right?
In any case, I think if we are to support 3+ tiers I have to change the
semantics to distinguish between higher and lower tiers instead to be
generic anyways, so I will change the wording to reflect this!

> > Next for the 3+ tier case, I think this is a lot more scalable than it seems
> > at first. This series depends on another RFC that I sent out [1],
> > which pushes the concept of "stock" from memcg to page_counter, which means
> > that it is more scalable to just add more page counters to each memcg.
> > This means that each tier would just need another page_counter to track its
> > memory usage, and we trigger selective reclaim on the tier that is being
> > targeted via the scan control nodemask we introduce in this series.
> > 
> > At my talk in LSFMMBPF, Usama noted that the user-visible API should probably
> > remain the same, no matter what. The way I have currently established the
> > memcg files aren't really scalable, so Usama suggested turning the
> > "memory.toptier_XXX" sysfs files to "memory.tiered_XXX", which would include
> > a newline-separated / space-separated list of per-tier limits. Something like:
> > 
> > $ cat memory.tiered_max
> > tier_0 20971520
> > tier_1 31457280
> > ...
> 
> memcgs are currently not NUMA aware, so this could be one possible direction.
> 
> But is "tier" the right abstraction? I really don't know.

>From my Pov after working on this for a while, I think tiering is the grouping
that makes the most sense. It is my understanding that memory present in
different nodes, but belonging to the same memcg are fungible; memory from 
one node isn't more or less scarce than the other node.

However, if the system has multiple tiers then the memory is no longer
fungible, and memcgs will be competing for the same scarce resource. So I think
it makes sense to limit how much a memcg can make.

> Are tier numbers really what we want? I don't know.

If this doesn't make sense, I can change this to just tiers 0-3 (MAX_NR_TIERS
is currently hardcoded to 4 across all architectures).

> We'd really need some feedback from memcg folks.

That would really be awesome : -)

> > So we have a way to make both the user-facing side stable, and the internals
> > also more scalable.
> > 
> > With that said, I've opted to leave the internals to 2 tiers for now -- I think
> > it is not too late to add the generalization series when we start seeing
> > 3+ tier systems out there in the wild. My goal was to introduce tieredness,
> > and we can work towards generalization in a future work.
> > 
> > On that note, it seems like in general mm is aware of 3+ tiers, but most of the
> > existing work revolves around distinguishing between toptier/everything else.
> > I got this impression from reading mm/memory-tiers.c -- but please feel free
> > to correct me if you feel like I have the wrong idea here : -)
> 
> I assume you're right, but it's something different if we end up providing a
> proper memcg API towards the users.
> 
> > So perhaps the generalization work would benefit from first introducing more
> > general tier awareness (not just toptier vs. rest) in memory-tiers.c.
> > 
> > What do you think? Does this approach of introducing toptier restriction for
> > now, and then generalizing in future work make sense to you?
> 
> I'd suspect that we'd want to support multiple tiers from day one -- especially
> get the interfaces right.

So I think the interface that I have at the moment (in the v3) is stable across
any number of memory tiers. My idea was to land something with a stable API
and then as future work support 3+ tiers. But if it makes more sense to land
the scalable tiered memcg work as the first iteration then I can work on that
too.

I've been working on v3 for a while, I will send it up sometime soon so that
I can start the discussion on this again. I really want to discuss with memcg
maintianers and migration maintainers to get some confirmation on whether
the series's direction looks good and whether they are happy with my
vision for the series.

And then in v4, I will add the scalable tiers.

How does that sound? : -) Again, thank you for coming back to this.
It means a lot!
Joshua