Re: [RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports

[Lian Wang (ProcessMission)]

Hi SJ,

Thank you for the questions.  I mixed the original field scenario and our
controlled reproducer in my previous mail.  I also jumped from the observation
problem to PMD mapping demotion before explaining why the deployment needs both
huge pages and finer-grained observation.  That made it sound as if our goal
was simply to stop using huge pages.

> The observation makes sense.  However, does the sparse access pattern
> realistic?  If so, what is the purpose or expected benefit of using huge page
> for workloads having such access pattern?

The original case is not a lab-created request to use huge pages for a sparse
microbenchmark.  It was reported by SXF from a KVM/QEMU deployment where the
VM has a large memory allocation backed by a shared tmpfs file, host THP is
enabled, and Oracle runs inside the guest.  They use host-side DAMON to measure
the VM's hot-memory proportion.  With the same business memory use, they
observed a much larger hot proportion when THP was enabled.

The guest tmpfs 4 KiB/2 MiB writer is their controlled diagnostic case for
isolating that observation; it is not a claim that the Oracle workload is
exactly a one-page-per-2-MiB loop.  We then reproduced and quantified the same
effect on a PC and on the x86 server.

For this deployment, keeping large pages is a real requirement at the host
virtualization layer.  QEMU owns a large resident guest-RAM mapping.  Host THP
allows that mapping to use PMDs and allows KVM to use large secondary mappings,
reducing host page-table memory, TLB and nested-page-walk pressure, and KVM
mapping/fault overhead.  These benefits are independent of whether every 4 KiB
page inside a particular 2 MiB range is hot at one observation time.

This host configuration should also not be confused with the guest database
page policy.  Oracle may separately use explicit HugePages for its SGA inside
the guest; the guest policy and the host THP backing of QEMU RAM are different
translation layers.  The requirement reported to us is to retain the host
large-page benefit while measuring the guest working-set proportion from the
host.  Disabling host THP merely to make DAMON's number smaller would change
the deployed VM configuration and remove the benefit the user is trying to
keep.

The monitoring requirement is different: DAMON is expected to estimate how
much of the guest memory is actually hot.  A host PMD is a translation unit for
guest RAM, not a semantic hotness unit for Oracle.  Guest allocation can place
small accessed pages across many guest-physical 2 MiB ranges, and one access
then makes each corresponding coarse mapping look accessed.

Our latest capture makes this mechanism concrete.  The sparse diagnostic
touched 24,576 guest 4 KiB pages spread over 24,575 guest-physical 2 MiB
buckets, while the dense control touched 12,582,912 pages.  They are 0.146%
and 75% of the monitored 64 GiB backend, respectively, but host DAMON reported
about 75% for both.  This is why the user needs huge-page mapping for VM
performance and, independently, finer-grained observation for a meaningful
hot-memory ratio.

The performance requirement and monitoring use case come from the deployment,
not from our lab model.  We will also report the measured Oracle/VM benefit and
the production access distribution when those data are ready.  The diagnostic
result establishes the observation mechanism, but it should not substitute for
those workload-level measurements.

> I'm not very sure if this is the right direction.  PMD mapping demotion sounds
> like you just don't want to use huge pages.  If so, you could disable huge
> pages.

I agree.  My reasoning in the previous mail was too jumpy: I went from a coarse
observation directly to a possible MM response, and that obscured the actual
goal.  PMD mapping demotion is not part of our current proposed solution.  The
current direction is to keep both the huge folio and PMD mapping unchanged and
use a genuinely fine-grained access primitive to improve the observation and
the stat-only decision.  We should first find out whether that is sufficient
before discussing any mapping change at all.

> technically speaking, it is not the report semantics.  Reporting allows any
> information to be reported.  Page faults like information could be coarse
> grained, same to the current page table accessed-bit based one.  Only finer
> grained access primitive reports, like those from perf events, would increase
> the accuracy for the sparse access pattern monitoring.

Agreed.  The report interface is the transport, not the source of accuracy.
Our recorded-address replay is intended only to test whether the DAMON
consumer and decision can use complete fine-grained address evidence.  The
eventual accuracy has to come from a fine-grained primitive such as an
appropriate perf event source, with its coverage and loss accounted for.  We
will use that terminology in the follow-up results.

With this context clarified, we will return to the immediate work: use the
existing IBS/perf-event proposal as the concrete strategy under test, and see
whether its fine-grained evidence can improve the observation and DAMOS
decision for this case.  We will use the results to test and review this series,
report both improvements and remaining gaps, and not assume in advance that it
is the final solution.

I hope this clarifies why keeping huge pages and requesting finer-grained
observation are not contradictory requirements in this case.

If I am still misunderstanding any part of your questions, or if any part of
this explanation remains unclear, please continue to correct me.  We will keep
sharing our findings and would like to make sure we are aligned before going
further.

Thanks,
Lian