Re: [PATCH 00/18] Another attempt at HVO support on arm64

[Dev Jain]

On 13/07/26 1:11 pm, Muchun Song wrote:
> 
> 
>> On Jul 13, 2026, at 12:59, James Houghton <jthoughton@xxxxxxxxxx> wrote:
>>
>> On Thu, Jul 9, 2026 at 9:59 PM Muchun Song <muchun.song@xxxxxxxxx> wrote:
>>>> On Jul 8, 2026, at 11:11, James Houghton <jthoughton@xxxxxxxxxx> wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> This patch series uses a trick with the Access Flag on CPUs that support
>>>> hardware update of the AF to update vmemmap page table entries without
>>>> introducing a time window where CPUs accessing the vmemmap might fault.
>>>
>>> Hi James,
>>>
>>> I was just looking into the Arm ARM specification regarding the TLB caching
>>> behavior to better understand the "AF trick" used here, and came across a few
>>> sections that got me a bit confused.
>>>
>>> Specifically, Section D8.5.1 [1] states Rule RDWZCQ:
>>>
>>>        "Descriptors with AF set to zero can never be cached in a TLB. For more
>>>         information about when translation table entries are permitted to be
>>>         cached in a TLB, see Translation Lookaside Buffers."
>>>
>>> This indeed seems to support the core assumption of the AF trick. However, when
>>> following the reference to Section D8.16 "Translation Lookaside Buffers" [2], it
>>> defines the following rule for permitted caching (RSQBCS):
>>>
>>>        "When address translation is enabled, a translation table entry for an
>>>         in-context translation regime that does not cause a Translation fault,
>>>         an Address size fault, or an Access flag fault is permitted to be cached
>>>         in a TLB or intermediate TLB caching structure as the result of an
>>>         explicit or speculative access."
>>>
>>> This is exactly where my confusion lies, as these two descriptions in the spec
>>> feel directly contradictory when FEAT_HAF(hardware management of the Access flag)
>>> is enabled.
>>
>> I don't think it intends to say that "translation table entries with
>> AF=0 may be cached" just because they don't cause Access flag faults.
>>
>> It is likely that this clause was written before FEAT_HAFDBS/FEAT_HAF
>> existed. Re-reading it with that in mind makes the meaning clearer:
>> PTEs with AF=1 may be cached if the architecture otherwise allows it.
>>
>> There is another clause, in D8.5 "Hardware updates of the translation
>> tables", I_RGQLZ: "If hardware does not update the AF from 0 to 1,
>> then the descriptor is not permitted to be cached in a TLB." I think
>> this supports my interpretation.
>>
>> I am convinced that it is impossible for an implementation not to
>> conform to Rule R_DWZCQ (PTEs with AF=0 cannot be cached) while
>> supporting HW AF correctly. Let me try to explain why.
>>
>> For HW AF to function correctly:
>> 1. Setting AF=1 must be done atomically, otherwise hardware could set
>> a bit in an invalid PTE. (This would break Linux swp_entrys.)
>> 2. The AF=0 translation cannot be used, otherwise we could
>> potentially drop AF updates.
>>
>> If an implementation used the translation before coherently setting
>> AF=1 to perform an architectural memory access, then it must choose
>> either to set AF=1 on a PTE that may have since changed (perhaps to an
>> invalid PTE) or drop the AF update entirely. Either way HW AF is
>> broken.
>>
>> Please also see I_GXGPN: "When the translation of an architecturally
>> executed memory access occurs, the architecture requires that AF is
>> set to 1."
> 
> Thanks for your excellent explanation.
> 
>>
>>> On one hand, Section D8.5.1 [1] explicitly uses the word "never" for AF=0
>>> descriptors. On the other hand, under FEAT_HAF, an entry with AF=0 does not cause
>>> an Access flag fault anymore, because the hardware is capable of updating the AF
>>> automatically. This technically makes it eligible for speculative caching under
>>> Rule RSQBCS in Section D8.16 [2].
>>>
>>> My question is: which part of the spec reflects the actual architectural intent
>>> when FEAT_HAF is enabled? Is an entry with AF=0 permitted to be speculatively
>>> cached in the TLB by the hardware under FEAT_HAF, or does the "never" in RDWZCQ
>>> still hold absolute priority?
>>>
>>> Please let me know if I have misread the specification or missed some overriding
>>> constraints here.
>>
>> Thanks for making me look a little harder into this; it's a good
>> question. I hope my reply above makes you more comfortable with the AF
> 
> At least to me, what you're saying makes sense.
> 
>> trick. I also hope to hear more from the Arm folks as well.
> 
> Yes, it would be best to have an ARM expert here to confirm that our
> understanding of the manual is completely accurate.
> 
> Thanks.
> 
>>
>> Thanks Muchun!
> 

Disclaimer: *NOT* an Arm expert, but it would also be good to prove the conjecture:

"If we guarantee that there is no caching structure holding a stale translation, then
we do not require BBM". Meaning, the only reason we do BBM is to prevent two execution
contexts from seeing two different translations, and such a situation can *only* happen
via TLB or intermediate caching structures.

D8.17.1, rule I_THWDH says:

"Use of a break-before-make sequence to ensure that old and new translation table entries
are never simultaneously visible to different execution threads is guaranteed to prevent
all of the following problems:

Creating multiple TLB entries that apply to the same address.

The effects of TLB caching breaking coherency.

The effects of TLB caching breaking single-copy atomicity properties.

The effects of TLB caching breaking ordering guarantees or uniprocessor semantics.

The effects of TLB caching causing a failure to clear the Exclusives monitors."


All these problems are derived from "TLB caching", so the conjecture seems to be true.

> 
>