[LSF/MM/BPF TOPIC] Unifying sources of page temperature information - what info is actually wanted?

[Jonathan Cameron]

On Fri, 31 Jan 2025 12:28:03 +0000
Jonathan Cameron <Jonathan.Cameron@xxxxxxxxxx> wrote:

> > Here is the list of potential discussion points:  
> ...
> 
> > 2. Possibility of maintaining single source of truth for page hotness that would
> > maintain hot page information from multiple sources and let other sub-systems
> > use that info.  
> Hi,
> 
> I was thinking of proposing a separate topic on a single source of hotness,
> but this question covers it so I'll add some thoughts here instead.
> I think we are very early, but sharing some experience and thoughts in a
> session may be useful.

Thinking more on this over lunch, I think it is worth calling this out as a
potential session topic in it's own right rather than trying to find
time within other sessions.  Hence the title change.

I think a session would start with a brief listing of the temperature sources
we have and those on the horizon to motivate what we are unifying, then
discussion to focus on need for such a unification + requirements 
(maybe with a straw man).

> 
> What do the other subsystems that want to use a single source of page hotness
> want to be able to find out? (subject to filters like memory range, process etc)
> 
> A) How hot is page X?  
> - Is this useful, or too much data? What would use it?
>   * Application optimization maybe. Very handy for developing algorithms
>     to do the rest of the options here as an Oracle!
> - Provides both the cold and hot end of the scale, but maybe measurement
>   techniques vary and can not be easily combined. Hard in general to combine
>   multiple sources of truth if aiming for an absolute number.
> 
> B) Which pages are super hot?
> - Probably these that make the most difference if they are in a slower memory tier.
> 
> C) Some pages are hot enough to consider moving?
> - This may be good enough to get the key data into the fast memory over time.
> - Can combine sources of info as being able to compare precise numbers doesn't matter.
> 
> D) Which pages are fairly cold?
> - Likewise maybe good enough over time.
> 
> E) Which pages are very cold?
> - Ideal case for tiering. Swap these with the super hot ones.
> - Maybe extra signal for swap / zswap etc
> 
> F) Did these hot pages remain hot (and same for cold)
> - This is needed to know when to back off doing things as we have unstable
>   hotness (two phase applications are a pain for this), sampling a few
>   pages may be fine.
> 
> Messy corners:
> 
> Temporal aspects.
> - If only providing lists of hottest / coldest in last second, very hard
>   to find those that are of a stable temperature. We end up moving
>   very hot data (which is disruptive) and it doesn't stay hot.
> - Can reduce that affect by long sampling windows on some measurement approaches
>   (on hardware trackers that can trash accuracy due to resource exhaustion
>    and other subtle effects).
> - bistable / phase based applications are a pain but perhaps up to higher
>   levels to back off.
> 
> My main interest is migrating in tiered systems but good to look at what
> else would use a common layer.
> 
> Mostly I want to know something that is useful to move, and assume convergence
> over the long term with the best things to move so to me the ideal layer has
> following interface (strawman so shoot holes in it!):
> 
> 1) Give me up to X hotish pages from a slow tier (greater than a specific measure
> of temperature)
> 2) Give me X coldish pages a faster tier.
> 3) I expect to ask again in X seconds so please have some info ready for me!
> 4) (a path to get an idea of 'unhelpful moves' from earlier iterations - this
>     is bleeding the tiering application into a shared interface though).
> 
> If we have multiple subsystems using the data we will need to resolve their
> conflicting demands to generate good enough data with appropriate overhead.
> 
> I'd also like a virtualized solution for case of hardware PA trackers (what
> I have with CXL Hotness Monitoring Units) and classic memory pool / stranding
> avoidance case where the VM is the right entity to make migration decisions.
> Making that interface convey what the kernel is going to use would be an
> efficient option. I'd like to hide how the sausage was made from the VM.
> 
> Jonathan
>