Re: [PATCH v3 0/4] Implement IOCTL to get and clear soft dirty PTE

[Muhammad Usama Anjum]

On 10/18/22 4:11 PM, Michał Mirosław wrote:
> On Tue, 18 Oct 2022 at 12:36, Muhammad Usama Anjum
> <usama.anjum@xxxxxxxxxxxxx> wrote:
> [...]
> > I've included the masks which the CRIU developers have specified.
> > max_out_page is another new optional variable which is needed to
> > terminate the operation without visiting all the pages after finding the
> > max_out_page number of desired pages. There is no way to terminate the
> > operation without this variable.
> >
> > How does the interface looks now? Please comment.
> >
> > /* PAGEMAP IOCTL */
> > #define PAGEMAP_GET             _IOWR('f', 16, struct pagemap_sd_args)
> > #define PAGEMAP_CLEAR           _IOWR('f', 17, struct pagemap_sd_args)
> > #define PAGEMAP_GET_AND_CLEAR   _IOWR('f', 18, struct pagemap_sd_args)
>
> Why are three IOCTLs needed? Could CLEAR be a flag (like the
> PAGEMAP_NO_REUSED_REGIONS) or 'cmask' and GET be implied when vec !=
> NULL?
Makes sense. Yes, this can be done.

> > /* Bits are set in the bitmap of the page_region and masks in
> > pagemap_sd_args */
> > #define PAGE_IS_SD      1 << 0
> > #define PAGE_IS_FILE    1 << 1
> > #define PAGE_IS_PRESENT 1 << 2
> > #define PAGE_IS_SWAPED  1 << 3
> >
> > /**
> >    * struct page_region - Page region with bitmap flags
> >    * @start:     Start of the region
> >    * @len:       Length of the region
> >    * bitmap:     Bits sets for the region
> >    */
> > struct page_region {
> >          __u64 start;
> >          __u64 len;
> >          __u64 bitmap;
> > };
>
> Could you explain what units start and len are using? Are they bytes
> or pages (what size)?
These are page sizes. Start must be the starting address of the page. 
Length don't need to be the exact multiple of the page size. All the 
pages in the length are included just like mincore() syscall.

> > /**
> >    * struct pagemap_sd_args - Soft-dirty IOCTL argument
>
> Nit: it's not soft-dirty-specific anymore. Maybe "pagemap_scan_args"?
>
> >    * @start:             Starting address
> >    * @len:               Length of the region
> >    * @vec:               Output page_region struct array
> >    * @vec_len:           Length of the page_region struct array
> >    * @max_out_page:      Optional max output pages (It must be less than
> > vec_len if specified)
>
> Why is it required to be less than vec_len? vec_len effectively
> specifies max number of ranges to find, and this new additional field
> counts pages, I suppose?
> BTW, if we count pages, then what size of them? Maybe using bytes
> (matching start/len fields) would be more consistent?
Yes, it if for counting pages. As the regions can have multiple pages, 
user cannot specify through the number of regions that how many pages 
does he need. Page size is used here as well like the start and len. 
This is optional argument as this is only needed to emulate the Windows 
syscall getWriteWatch.

> >    * @flags:             Special flags for the IOCTL
> >    * @rmask:             Special flags for the IOCTL
> >    * @amask:             Special flags for the IOCTL
> >    * @emask:             Special flags for the IOCTL
> >    * @__reserved:                Reserved member to preserve data alignment. Must be 0.
> >    */
> > struct pagemap_sd_args {
> >          __u64 __user start;
> >          __u64 len;
> >          __u64 __user vec; // page_region
> >          __u64 vec_len;    // sizeof(page_region)
> >          __u32 flags;      // special flags
> >          __u32 rmask;
> >          __u32 amask;
> >          __u32 emask;
> >          __u32 max_out_page;
> >          __u32 __reserved;
> > };
> >
> > /* Special flags */
> > #define PAGEMAP_NO_REUSED_REGIONS       0x1
>
> What does this flag do?
Some non-dirty pages get marked as dirty because of the kernel's 
internal activity. The dirty bit of the pages is stored in the VMA flags 
and in the per page flags. If any of these two bits are set, the page is 
considered to be dirty. Suppose you have cleared the dirty bit of half 
of VMA which will be done by splitting the VMA and clearing dirty flag 
in the half VMA and the pages in it. Now kernel may decide to merge the 
VMAs again as dirty bit of VMAs isn't considered if the VMAs should be 
merged. So the half VMA becomes dirty again. This splitting/merging 
costs performance. The application receives a lot of pages which aren't 
dirty in reality but marked as dirty. Performance is lost again here.

This PAGEMAP_NO_REUSED_REGIONS flag is used to don't depend on the dirty 
flag in the VMA flags. It only depends on the individual page dirty bit. 
With doing this, the new memory regions which are just created, doesn't 
look like dirty when seen with the IOCTL, but look dirty when seen from 
pagemap. This seems okay as the user of this flag know the implication 
of using it.

> Best Regards
> Michał Mirosław