Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure

[Gregory Price]

On Mon, Aug 10, 2026 at 05:16:39AM +0100, Matthew Wilcox wrote:
> On Mon, Aug 10, 2026 at 11:38:14AM +0800, Yongting Lin wrote:
> > On Tue, Jul 28, 2026 at 07:24:46PM +0100, Matthew Wilcox wrote:
> > 
> > From our perspective, this is not merely a hypothetical use case. We
> > are actively developing and evaluating CXL tiered-memory systems, and
> > expect promoting CXL-resident pages that become hot back to DRAM to be
> > a practical requirement.
> 
> But what *is* your use case?  Truly hot data ends up in L1/L2/L3 no
> matter whether it came across the ridiculously high latency CXL link or
> from regular DRAM.  So we're talking about "warm" data that's presumably
> measured in gigabytes (since current server CPUs have about half a
> gigabyte of LLC)
> 

Speaking for myself (not Yongting Lin) - our use case is pretty straight
forward and deployed at scale today, with plans to expand on it in
future generations.

We have systems with 1TB+ of memory, with a 768GB/256GB DRAM/CXL split.

We're expecting these numbers to grow, along with density of work.

On these systems we use both scaled and stacked workloads (single and
multi-container) where we do have a decent amount of warm data landing
on CXL. In fact, we're looking to make that a more explicit scenario
to reduce per-workload variance (higher floors, lower ceilings).

See Joshua's Tiered Memcg Limits work [0]
[0] https://lore.kernel.org/all/20260807202059.2620949-1-joshua.hahnjy@xxxxxxxxx/

Right now this works when mixing workloads which largely use mapped
memory, but for workloads using unmapped pagecache, they end up
permanently demoted to CXL because NUMA Balancing can't promote page
cache (but reclaim can demote it, and the page allocator can fallback
to CXL).

PGHot at least gives us the *full* solution that NUMA balancing does not.

> I see the basis for saying "this task is low priority, it gets its memory
> allocated on CXL" and "this task is high priority, it gets its memory
> allocated on DRAM".  I don't see the use case where we're trying to figure
> out that region A of this task is sufficiently warmer than region B, and so
> we want to demote region B back to CXL and promote region A back to DRAM.
> 

We find the vast, vast majority of "low priority" work ends up pushed
out to swap / fully invalidated anyway, and not even taking up much of
any memory in the first place.  This work is not the problem - it's
usually things like system updates or monitors etc.

Actual user work wanting X-GB of memory is almost never willing to
take the latency hit of 100% CXL - and even if you did that, you
actually cause bandwidth issues for the rest of the system due to
limited bandwidth compared to local DRAM.

Something to remember: Demotion does not respect cpusets or mempolicy,
and page cache is owned by the first-toucher, so cpuset limitations
break for page cache in general.  (This is one of many reasons I've
been exploring the private nodes series).

===

For actual tangible problems: 

We've found fallback allocations landing on CXL is preferable to forcing
reclaim to run and demote to CXL before a new DRAM allocation can be
made. e.g. zone_reclaim_mode=0x0 is better than 0xf.  ZRM induced stalls
are just always worse than eating a CXL page and moving it later.

This directly results in hot memory (in fact many gbs of it) on CXL.

This is especially prevelent when workloads spinup after the top tier
is already / near full.  The tier-aware memcg stuff should help with
this, but it still requires both demotion and promotion to level out
the usage between tiers to an acceptable ratio.

I'm not looking for a system that can predict the future and maximize
performance.  At best I'd like a system that nudges placement in the
right direction over a long period for better workload consistency.

> I particularly don't see a case for trying to do it on single-page
> granularity.  There's just too much damn information to track.  Maybe at
> a 2MB or 1GB boundary, but not per page.

I largely agree with you here.  If you look at the other work my
colleagues have been doing - it's been heavily focused on making
THPs more reliable and even pushing towards 1GB THPs.  We're getting
to a point where 4kb just doesn't make sense anymore for many things.

~Gregory