Re: memcg reclaim demotion wrt. isolation

[Michal Hocko]

On Wed 14-12-22 13:40:33, Johannes Weiner wrote:
> On Wed, Dec 14, 2022 at 10:42:56AM +0100, Michal Hocko wrote:
[...]
> > The reclaim behavior would make more sense to me if it was constrained
> > to the allocating memcg hierarchy so unrelated lruvecs wouldn't be
> > disrupted.
> 
> What if the second tier is full, and the memcg you're trying to demote
> doesn't have any pages to vacate on that tier yet? Will it fail to
> demote?
> 
> Does that mean that a shared second tier node is only usable for the
> cgroup that demotes to it first? And demotion stops for everybody else
> until that cgroup vacates the node voluntarily?
> 
> As you can see, these would be unprecedented and quite surprising
> first-come-first-serve memory protection semantics.

This is a very good example!

> The only way to prevent cgroups from disrupting each other on NUMA
> nodes is NUMA constraints. Cgroup per-node limits. That shields not
> only from demotion, but also from DoS-mbinding, or aggressive
> promotion. All of these can result in some form of premature
> reclaim/demotion, proactive demotion isn't special in that way.

Any numa based balancing is a real challenge with memcg semantic. I do
not see per numa node memcg limits without a major overhaul of how we do
charging though. I am not sure this is on the table even long term.
Unless I am really missing something here we have to live with the
existing semantic for a foreseeable future.
 
> The default behavior for cgroups is that without limits or
> protections, resource access is unconstrained and competitive. Without
> NUMA constraints, it's very much expected that cgroups compete over
> nodes, and that the hottest pages win out. Per aging rules, freshly
> demoted pages are hotter than anything else on the target node, so it
> should displace accordingly.

That is certainly a way to look at it but I would really emphasise
that this competition depends quite significantly on a higher level
balancing on top. Memory allocations fall back to different nodes so the
resource distribution should be roughly even in this case. If there is a
competition then it most likely means our resources are overcommitted.

The picture is slightly different with the demotion for memory tiering
IMHO because that spills an internal resource contention or explicit
user space balancing (via pro-active reclaim/demotion) outside because
it creates pressure on the demotion target that is a shared resource as
you have mentioned above.
 
> Consider the case where you have two lower tier nodes and there are
> cpuset isolation for the main workloads, but some maintenance thing
> runs and pollutes one of the lower tier nodes.

Well, this is not really much different from regular NUMA system where
node aware and constrained workloads compete with NUMA unconstrained
workloads. This has never worked.

> Or consider the case
> where a shared lower tier node is divvied up between two cgroups using
> protection settings to allow overcommit, i.e. per-node memory.low.
 
> Demotions, proactive or not, MUST do global reclaim on a full node.

OK, but my concern is how to implement any usersoace policy around that
behavior. If you see demotion failures then you can trigger some
rebalancing explicitly. If those are silent then your only option left
is to check the capacity of the demotion target regularly and play a
catch up game. Is this sufficient?

All that being said, I can see that both approaches result in some
corner cases. I do agree that a starvation is likely easier scenario
than an actively evil container disrupting another container by pushing
its demoted pages out. 

So scratch the patch.

Thanks
-- 
Michal Hocko
SUSE Labs