Re: [PATCH v11 0/7] Implement IOCTL to get and optionally clear info about PTEs

[Muhammad Usama Anjum]

Hi Andrew,

Thank you for your message.

On 3/10/23 12:58 AM, Andrew Morton wrote:
> On Thu,  9 Mar 2023 18:57:11 +0500 Muhammad Usama Anjum <usama.anjum@xxxxxxxxxxxxx> wrote:
> 
>> The information related to pages if the page is file mapped, present and
>> swapped is required for the CRIU project [5][6]. The addition of the
>> required mask, any mask, excluded mask and return masks are also required
>> for the CRIU project [5].
> 
> It's a ton of new code and what I'm not seeing in here (might have
> missed it?) is a clear statement of the value of this feature to our
> users.
Sorry, let me give some clear details about its value.

> 
> I see hints that CRIU would like it, but no description of how valuable
> this is to CRIU's users.
> 
> So please spend some time preparing this info.
> 
> Also, are any other applications of this feature anticipated?  If so,
> what are they?
> 
> IOW, please sell this stuff to us!
The real motivation for adding PAGEMAP_SCAN IOCTL is to emulate Windows
GetWriteWatch() syscall [1].
> The GetWriteWatch{} retrieves the addresses of the pages that are written
> to in a region of virtual memory.
This syscall is used in Windows applications and games etc. This syscall is
being emulated in pretty slow manner in userspace. Our purpose is to
enhance the kernel such that we translate it efficiently in a better way.
Currently some out of tree hack patches are being used to efficiently
emulate it in some kernels. We intend to replace those with these patches.
So the whole gaming on Linux can effectively get benefit from this. It
means there would be tons of users of this code.

CRIU use case [2] was mentioned by Andrei and Danylo:
> Use cases for migrating sparse VMAs are binaries sanitized with ASAN,
> MSAN or TSAN [3]. All of these sanitizers produce sparse mappings of
> shadow memory [4]. Being able to migrate such binaries allows to highly
> reduce the amount of work needed to identify and fix post-migration
> crashes, which happen constantly.
At [10]:
> For Andrei's use case is to iteratively dump memory.

@Andrei and Danylo can elaborate more on their use cases.


*Implementation Evolution (Short Summary)*
>From the definition of GetWriteWatch(), we feel like kernel's soft-dirty
feature can be used under the hood with some additions like:
* reset soft-dirty flag for only a specific region of memory instead of
clearing the flag for the entire process
* get and clear soft-dirty flag for a specific region atomically

So we decided to use ioctl on pagemap file to read or/and reset soft-dirty
flag. But using soft-dirty flag, sometimes we get extra pages which weren't
even written. They had become soft-dirty because of VMA merging and
VM_SOFTDIRTY flag. This breaks the definition of GetWriteWatch(). We were
able to by-pass this short coming by ignoring VM_SOFTDIRTY until David
reported that mprotect etc messes up the soft-dirty flag while ignoring
VM_SOFTDIRTY [5]. This wasn't happening until [6] got introduced. We
discussed if we can revert these patches. But we could not reach to any
conclusion. So at this point, I made couple of tries to solve this whole
VM_SOFTDIRTY issue by correcting the soft-dirty implementation:
* [7] Correct the bug fixed wrongly back in 2014. It had potential to cause
regression. We left it behind.
* [8] Keep a list of soft-dirty part of a VMA across splits and merges. I
got the reply don't increase the size of the VMA by 8 bytes.

At this point, we left soft-dirty considering it is too much delicate and
userfaultfd [9] seemed like the only way forward. From there onward, we
have been basing soft-dirty emulation on userfaultfd wp feature where
kernel resolves the faults itself when WP_ASYNC feature is used. It was
straight forward to add WP_ASYNC feature in userfautlfd. Now we get only
those pages dirty or written-to which are really written in reality. (PS
There is another WP_UNPOPULATED userfautfd feature is required which is
needed to avoid pre-faulting memory before write-protecting [9].)

All the different masks were added on the request of CRIU devs to create
interface more generic and better.

[1]
https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getwritewatch
[2] https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@xxxxxxxxxx
[3] https://github.com/google/sanitizers	
[4] https://github.com/google/sanitizers/wiki/AddressSanitizerAlgorithm#64-bit
[5]
https://lore.kernel.org/all/bfcae708-db21-04b4-0bbe-712badd03071@xxxxxxxxxx
[6] https://lore.kernel.org/all/20220725142048.30450-1-peterx@xxxxxxxxxx/
[7]
https://lore.kernel.org/all/20221122115007.2787017-1-usama.anjum@xxxxxxxxxxxxx
[8]
https://lore.kernel.org/all/20221220162606.1595355-1-usama.anjum@xxxxxxxxxxxxx
[9] https://lore.kernel.org/all/20230306213925.617814-1-peterx@xxxxxxxxxx
[10] https://lore.kernel.org/all/20230125144529.1630917-1-mdanylo@xxxxxxxxxx

-- 
BR,
Muhammad Usama Anjum