Re: [PATCH v1 0/5] arm64/mm: uffd write-protect and soft-dirty tracking

[Shivansh Vij]

(Sorry about the previous HTML email, accidentally used the wrong email client) 

Hey All,

>On 19/04/2024 08:43, Ryan Roberts wrote:
>> Hi All,
>>
>> This series adds uffd write-protect and soft-dirty tracking support for arm64. I
>> consider the soft-dirty support (patches 3 and 4) as RFC - see rationale below.
>>
>> Previous attempts to add these features have failed because of a perceived lack
>> of available PTE SW bits. However it actually turns out that there are 2
>> available but they are hidden. PTE_PROT_NONE was previously occupying a SW bit,
>> but it only applies when PTE_VALID is clear, so this is moved to overlay PTE_UXN
>> in patch 1, freeing up the SW bit. Bit 63 is marked as "IGNORED" in the Arm ARM,
>> but it does not currently indicate "reserved for SW use" like it does for the
>> other SW bits. I've confirmed with the spec owner that this is an oversight; the
>> bit is intended to be reserved for SW use and the spec will clarify this in a
>> future update.
>>
>> So we have our two bits; patch 2 enables uffd-wp, patch 3 enables soft-dirty and
>> patches 4 and 5 sort out the selftests so that the soft-dirty tests are compiled
>> for, and run on arm64.
>>
>> That said, these are the last 2 SW bits and we may want to keep 1 bit in reserve
>> for future use. soft-dirty is only used for CRIU to my knowledge, and it is
>> thought that their use case could be solved with the more generic uffd-wp. So
>> unless somebody makes a clear case for the inclusion of soft-dirty support, we
>> are probably better off dropping patches 3 and 4 and keeping bit 63 for future
>> use. Although note that the most recent attempt to add soft-dirty for arm64 was
>> last month [1] so I'd like to give Shivansh Vij the opportunity to make the
>> case.
>
> Ugh, forgot to mention that this applies on top of v6.9-rc3, and all the uffd-wp
> and soft-dirty tests in the mm selftests suite run and pass. And no regressions
> are observed in any of the other selftests.

Appreciate the opportunity to provide input here. 

I personally don't know of any applications other than CRIU that make heavy use of soft-dirty, and my use case is specifically focused on adding live-migration support to CRIU on ARM. 

Cloud providers like AWS have pretty massive discounts for ARM-based spot instances (90% last time I checked), and having live-migration in CRIU would allow more applications to take advantage of that.

As Ryan mentioned, there are two ways to achieve this - add dirty tracking to ARM (Patch 3/4), or tear out the existing dirty tracking code in CRIU and replace it with uffd-wp. 

I picked option one (dirty tracking in arm) because it seems to be the simplest way to move forward, whereas it would be a relatively heavy effort to add uffd-wp support to CRIU. 

>From a performance perspective I am also a little worried that uffd will be slower than just tracking the dirty bits asynchronously with sw dirty, but maybe that's not as much of a concern with the addition of uffd-wp async. 

With all this being said, I'll defer to the wisdom of the crowd about which approach makes more sense - after all, with this patch we should get uffd-wp support on arm so at least there will be _a_ way forward for CRIU (albeit one requiring slightly more work). 

Thanks,
Shivansh