Re: [RFC PATCH 5/5] KVM: arm64: Enable HAFDBS for guests not on migration

[Tian Zheng]

On 10/27/2026 7:46 PM, Leonardo Bras wrote:
> On Tue, Oct 27, 2026 at 10:02:55AM +0800, Tian Zheng wrote:
> > On 10/23/2026 8:01 PM, Leonardo Bras wrote:
> > > On Fri, Oct 23, 2026 at 03:23:04PM +0800, Tian Zheng wrote:
> > > > On 10/14/2026 11:56 PM, Leonardo Bras wrote:
> > > > > On Thu, Sep 17, 2026 at 02:40:42PM +0100, Leonardo Bras wrote:
> > > > > > On Wed, Sep 16, 2026 at 04:27:19PM -0700, Oliver Upton wrote:
> > > > > > > On Wed, Sep 16, 2026 at 03:00:46PM +0100, Leonardo Bras wrote:
> > > > > > > > On Tue, Sep 15, 2026 at 05:10:02PM -0700, Oliver Upton wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > On Tue, Sep 01, 2026 at 06:15:56PM +0100, Leonardo Bras wrote:
> > > > > > > > > > When dirty-logging is disabled, even non-write faults make a page dirty,
> > > > > > > > > > which avoids a second fault when the page is actually written to.
> > > > > > > > > >
> > > > > > > > > > On dirty-logging enable, this approach causes all (writable) pages on the
> > > > > > > > > > memslot to be marked clean, even if they were not written to, which can
> > > > > > > > > > take a lot of time, while holding the MMU lock, doing atomic writes to
> > > > > > > > > > PTEs.
> > > > > > > > >
> > > > > > > > > Do you have any performance numbers for this? Enabling HAFDBS seems a
> > > > > > > > > bit involved to avoid some stores on the first pass.
> > > > > > > >
> > > > > > > > Not yet, but if the idea does not look too crazy I can find hardware and
> > > > > > > > collect some data :)
> > > > > > >
> > > > > > > TBH this looks like a micro-optimization so I'm not expecting the
> > > > > > > performance gains to justify the behavior change.
> > > > > >
> > > > > > Okay then, I will get a machine to figure the performance gains :)
> > > > >
> > > > > Got the performance numbers here.
> > > > > I tested both the case where all PTEs are dirty, and the case that 50% of
> > > > > the PTEs are dirty.
> > > > >
> > > > > For that test, actually patch #5 could make kvm_pgtable_stage2_wrprotect()
> > > > > slower, which is counter-intuitive, in all platforms tested:
> > > > >
> > > > > When we walk the S2 pagetable to wrprotect, using the new walker, the
> > > > > write to the PTE itself is so cheap* that the cost of the branch
> > > > > misprediction (write PTE or not) for the 50/50 case is significantly higher
> > > > > than the cost of the 50% extra writes, which makes the 50/50 up to 60%
> > > > > slower than the 100% dirty case. It is still slower in the '25% dirty' as
> > > > > well, and starts to get even at '10% dirty'.
> > > > >
> > > > > * (non-atomic, everything is in order, walker is small, cpu will probably
> > > > > bundle it all in a single cacheline write)
> > > > >
> > > > > Since the idea of patch #5 was to reduce PTE writes in the first wrprotect
> > > > > walk, it becomes faster only if only a small portion (<10%) of the VM was
> > > > > write-faulted while the remaining was read-faulted, which seems to be a
> > > > > really reduced audience.
> > > > >
> > > > > Since this comes with increased complexity, and also carry costs of
> > > > > having HAFDBS itself enabled outside dirty-tracking, I don't think it's a
> > > > > good idea to further pursue patch #5. It's only value is that it served as
> > > > > a base for HDBSS implementation.
> > > > >
> > > > > Thanks for reading!
> > > > > Leo
> > > > Hi Leo,
> > > >
> > > > Thanks for running this, the numbers are really useful.
> > > >
> > > > It means the hardware dirty mode (HD=1) should only be enabled while
> > > > dirty logging is active with HDBSS support, and stays off otherwise.
> > >
> > > Hi Tian,
> > >
> > > Correct, it's actually counter-productive for it to stay on without HDBSS
> > > before dirty-logging.
> > >
> > > > With HD only enabled during dirty logging, we also keep the DBM-based
> > > > encoding for the HD=0 case. On systems that support FEAT_HAFDBS, DBM
> > > > remains the write bit and S2AP[1] the dirty bit even with HD=0. This
> > > > works because kvm_s2_fault_compute_prot() maps any writable page of a
> > > > non-tracked memslot writable-dirty (WD), including on read faults:
> > > >
> > > > ```c
> > > > if (s2vi->map_writable) {
> > > >           *prot |= KVM_PGTABLE_PROT_W;
> > > >
> > > >           if (s2vi->device ||
> > > >               !memslot_is_logging(s2fd->memslot) ||
> > > >               kvm_is_write_fault(s2fd->vcpu))
> > > >                   *prot |= KVM_PGTABLE_PROT_DIRTY;
> > > > }
> > > > ```
> > > >
> > > > So there is no WC entry for hardware to promote, and HD is simply not
> > > > needed outside dirty logging.
> > > >
> > > > In the next HDBSS version, we'll only turn on HD=1 when dirty logging
> > > > is active.
> > >
> > > Right!
> > > I will also send a new version of the pre-patch (4 patches) for us to
> > > discuss. The first 2 patches will change meaning a bit, but should still
> > > work as they do today.
> > >
> > > Thanks!
> > > Leo
> > Hi Leo,
> >
> > OK, I've seen the new DBM RFC patchset. HDBSS v6 will be based on it.
>
>
> Hi Tian, thanks for reading it, and for accepting it in your series!
>
> Before rebasing, please wait for the ongoing discussion on it's
> correctness. Maybe I will have to send new versions before it can be used.
>
> Thanks!
> Leo

Hi Leo,

I've read your discussion with Oliver on DBM RFC v2. As a suggestion,
maybe we should go with Oliver's approach, i.e. revert to the v1 scheme
and continue fixing from there.

Thanks,
Tian