Re: [PATCH v2 1/3] sched/numa: make free-space tiering promotion bypass configurable

[Li Zhe]

On 10/29/26 6:22 PM, Bharata B Rao wrote:
> On 29-Oct-26 3:16 PM, Li Zhe wrote:
>> [You don't often get email from lizhe.67@xxxxxxxxxxxxx. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> On 10/29/26 5:34 PM, Bharata B Rao wrote:
>>> On 28-Oct-26 9:42 AM, Li Zhe wrote:
>>>> Memory tiering normally promotes any recently accessed slow-tier page if
>>>> the target fast-tier node has enough free space. This preserves the
>>>> existing policy of filling available fast memory before relying on
>>>> hotness and rate-limit control.
>>>>
>>>> That is not always the desired policy. Some deployments may intentionally
>>>> keep part of a workload's memory on CXL-attached memory, for example to
>>>> preserve a desired DRAM/CXL capacity split or bandwidth distribution. With
>>>> the free-space bypass, short cold accesses and bursts of hot accesses can
>>>> refill the fast tier and pull the workload away from that distribution
>>>> whenever DRAM has free space.
>>>>
>>>> Redis testing exercises this policy issue. On a CXL system with node0 as
>>>> DRAM and node3 as slow-tier memory, a Redis test was run with
>>>> memory-tiering-only mode enabled. The test used a Redis dataset with 12
>>>> million 4KB values. It reclaimed memory until about 85% of Redis
>>>> anonymous memory was on node3, used a Redis benchmark configuration with
>>>> a 10% hot-key working set, and then issued a short 256MB access burst to
>>>> cold keys outside that working set. The cold-key burst mainly touches
>>>> memory outside the hot working set, which is expected to remain mostly on
>>>> the slow tier.
>>> Can you please clarify if 85% of redis memory ended up on lower tier naturally
>>> due to overflow from toptier? I ask this question because, if it is a natural
>>> overflow/demotion, then it will be hard to find head room in the toptier to
>>> allow promotions that bypass hotness or rate limit checks."
>>
>> It was not a natural overflow from a full top tier.
> Thanks for confirming this.
>
>> The test intentionally reclaimed/demoted Redis memory to the lower tier
>> to model a deployment that keeps only a limited fraction of the workload
>> in DRAM and leaves the rest on CXL memory.  In that setup the top tier can
>> still have enough free space, so the current free-space bypass can promote
>> short cold accesses back to DRAM.
> With pghot, I have been providing benchmark numbers from two scenarios:
>
> 1. Non-overcommitted case: The workload memory is pushed to lower tier before
> the measurement phase begins, the toptier has enough memory to accept the
> the incoming promotions.
>
> 2. Overcommitted case: The workload memory naturally overflows/demotes into
> the lower tier and every promotion undergoes the hotness + ratelimiting check.
>
> I gather that your measurement technique is similar to case 1 above.
>
> The feedback I have received for pghot is this: Non-overcommitted case is not
> the right methodology to use but the natural spillover is. Hence I am not sure
> if optimizing the free-space bypass case would benefit any naturally run
> cases.


Thanks for sharing this feedback.

Yes, the case I am trying to cover is closer to the non-overcommitted
case.  We do have deployments where memory is intentionally placed across
DRAM and CXL memory, instead of relying only on natural spillover from a
full top tier.

For example, the orchestration layer may want a cgroup to use a planned
DRAM/CXL ratio, so that DRAM capacity remains available for later
placement decisions or other workloads.

In such setups, the top tier can still have free space, but we may not
want short cold accesses to refill it simply because the free-space
bypass is available.  The intent of this patch is to let those
deployments keep using the existing hotness and rate-limit checks even
when there is top-tier headroom.

Thanks,
Zhe

>
> Regards,
> Bharata.