Re: [PATCH 00/18] Another attempt at HVO support on arm64

[James Houghton]

On Thu, Jul 9, 2026 at 8:41 PM Muchun Song <muchun.song@xxxxxxxxx> wrote:
>
> > On Jul 10, 2026, at 03:04, James Houghton <jthoughton@xxxxxxxxxx> wrote:
> >
> > On Thu, Jul 9, 2026 at 2:55 AM Muchun Song <muchun.song@xxxxxxxxx> wrote:
> >>> On Jul 9, 2026, at 00:49, James Houghton <jthoughton@xxxxxxxxxx> wrote:
> >>>
> >>> On Wed, Jul 8, 2026 at 1:41 AM Muchun Song <muchun.song@xxxxxxxxx> wrote:
> >>>> Do you mean that the support for AF might vary across different CPUs?
> >>>> I'm not that familiar with arm64, so it seems a bit strange to me that
> >>>> such basic hardware features can differ so much from one CPU to another.
> >>>
> >>> Yes, hardware updates of the Access Flag is a per-CPU feature. It is
> >>> available for a CPU to use if TCR_EL1.HA is set. TCR_EL1 is a system
> >>> register; each CPU has its own. (Linux will always enable HW AF for a
> >>> CPU when it is onlined[1] if support is advertised, so we simply need
> >>> to check if support is advertised to know that it is in fact enabled.)
> >>>
> >>> These days it is not uncommon for a system to have two (or more?)
> >>> different core implementations, like with "fast" cores and "efficient"
> >>> cores.
> >>>
> >>> [1] See the CONFIG_ARM64_HW_AFDBM bits in arch/arm64/mm/proc.S
> >>
> >> Thanks for your detailed explanation. When enabling HVO via the cmdline,
> >> can we simply prevent CPUs that do not support AF from coming online?
> >> Would implementing it this way be much simpler? In practice, developers
> >> definitely know whether the current system is suitable for enabling HVO.
> >> If some CPUs do not support AF, they would just need to evaluate the
> >> trade-off between memory savings and having fewer online CPUs than expected.
> >>
> >> For scenarios where HVO is enabled via sysctl, we simply need to check
> >> if all CPUs support AF. If any do not, the system should return an error.
> >>
> >> Then, we can proceed with the Pre-HVO.
> >
> > I don't think it makes sense to try to implement pre-HVO.
> >
> > We cannot do HVO if any boot CPUs do not support HW AF, as HW AF will
> > be required to free the HugeTLB pages later, which we should continue
> > to support. Pre-HVO (today anyway) happens before all boot CPUs are
> > onlined. IMO it is not okay to prevent boot CPUs from onlining.
>
> When the boot CPU starts, it first checks if the hardware (the current
> boot CPU) supports AF (The selection of the Boot CPU is critical here
> If the user really wants to enable HVO). If AF is supported and HVO is
> enabled via cmdline, we perform a Pre-HVO. Subsequently, any CPUs that
> do not support AF will be prevented from coming online.

I understand. I don't think we should do this. I think it's much
better to always be able to start all boot CPUs[1] and simply rely on
"normal" HVO.

[1] Just to be clear on terminology, I'm using "boot CPUs" to refer to
all CPUs that are onlined at boot-time, not just to *the* boot CPU
that starts the kernel.

>
> If the boot CPU does not support AF, enabling HVO is not permitted—neither
> via cmdline nor sysctl. All CPUs are allowed to be online.
>
> Once the system has started, there is only one way to enable HVO, and
> that is through sysctl. 1) If the current system includes any CPUs that do
> not support AF, enabling HVO is not permitted (unless the user chooses to
> take these CPUs that don't support AF offline). 2) If all online CPUs in the
> current system support AF, then HVO can be enabled. In this case, the
> system must also block any CPUs that do not support AF from coming online
> in the future.

An approach similar to this is implemented in this series: if there
are (or were) any HVOed folios, onlining a CPU without HW AF is not
allowed. If HW AF is not supported by any onlined CPU, HVO cannot
proceed (though it will appear to be enabled).

I probably should have integrated
`arch_hugetlb_vmemmap_optimization_supported()` and a `hvo_status`
check into the sysctl handler for HVO. I'll do this if it is still
relevant in v2.

> Of course, if a user specifically wants those CPUs to be
> allowed to come online, they can choose to disable HVO first (At the same time,
> this means that the HVO-optimized HugeTLB must be freed first as well).

I could make it so that, if there are no HVOed pages anymore, onlining
incompatible CPUs becomes permitted again. I don't really see the need
to do this.

> My proposal assumes that even though the system supports different types of
> CPUs, their support for AF (I suppose this is a very basic feature nowadays)
> is generally the same. I believe that for servers (Only servers stand to
> benefit more from HVO), there shouldn't be a mix of those that support AF
> and those that don't. At the very least, such a situation would be extremely
> uncommon. I believe this may be acceptable. That said, if we go this route,
> is there a way to simplify the code implementation even further?. Of course,
> this is just my personal speculation. Please let me know if I've missed
> anything.

I agree that systems where HVO is relevant are very unlikely to have
mixed support for HW AF.

I don't think the approach you're suggesting in this email (as I've
understood it) simplifies this series.

I don't want to mess with the arm64 cpufeature bits more than I've
done here; preventing boot CPUs from onlining will certainly be more
work. :) And then your other suggestions seem to be quite similar to
what is already in these patches.

I am happy to get rid of the "disable HVO if it is unused and a CPU
without support is onlined" part though and replace it with something
similar (the simplification from the other thread[2]).

[2] https://lore.kernel.org/linux-mm/674EC6D6-B1F7-447D-B462-861ECA5A99B7@xxxxxxxxx/