Re: [PATCH v2 2/2] mm: Remember young/dirty bit for page migrations

[Peter Xu]

On Tue, Aug 09, 2022 at 04:45:32PM +0800, Huang, Ying wrote:
> Peter Xu <peterx@xxxxxxxxxx> writes:
> 
> > On Thu, Aug 04, 2022 at 03:40:57PM -0700, Nadav Amit wrote:
> >> On Aug 4, 2022, at 1:39 PM, Peter Xu <peterx@xxxxxxxxxx> wrote:
> >> > +
> >> > static inline bool is_pfn_swap_entry(swp_entry_t entry);
> >> > 
> >> > /* Clear all flags but only keep swp_entry_t related information */
> >> > @@ -265,6 +285,57 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset)
> >> > 	return swp_entry(SWP_MIGRATION_WRITE, offset);
> >> > }
> >> > 
> >> > +/*
> >> > + * Returns whether the host has large enough swap offset field to support
> >> > + * carrying over pgtable A/D bits for page migrations.  The result is
> >> > + * pretty much arch specific.
> >> > + */
> >> > +static inline bool migration_entry_supports_ad(void)
> >> > +{
> >> > +	/*
> >> > +	 * max_swapfile_size() returns the max supported swp-offset plus 1.
> >> > +	 * We can support the migration A/D bits iff the pfn swap entry has
> >> > +	 * the offset large enough to cover all of them (PFN, A & D bits).
> >> > +	 */
> >> > +#ifdef CONFIG_SWAP
> >> > +	return max_swapfile_size() >= (1UL << SWP_MIG_TOTAL_BITS);
> >> 
> >> This is an actual a function call (unless LTO has some trick). A bit of a
> >> shame it cannot be at least memoized.
> >> 
> >> Or at least mark max_swapfile_size() as __attribute_const__ so it would not
> >> be called twice for make_migration_entry_young() and
> >> make_migration_entry_dirty().
> >
> > I didn't take too much effort on this one since we're on swap path and I
> > assumed that's not a super hot path.  But __attribute_const__ sounds good
> > and easy to get, thanks.
> >
> > Perhaps I should mark it on migration_entry_supports_ad() as a whole?  Note
> > that unfortunately SWP_MIG_TOTAL_BITS may not be a const either (see how
> > that define roots back to MAX_PHYSMEM_BITS, where on x86_64 it needs to
> > check 5-lvl).
> 
> I think it's possible to memorize max_swapfile_size() or
> migration_entry_supports_ad().  Although they are not constant, they are
> not changed after initialized.  The challenge is to find a clean way to
> initialize it.

I checked it up today, the conclusion is I think it's safe we initialize ad
bits for migration in swapfile_init() and I'll do that in the next version.

Longer proof material below.

Generic max_swapfile_size() is pretty much a constant except x86.  x86 has
two dependency, on (1) X86_BUG_L1TF, and (2) l1tf_mitigation.

The other challenge is the reference to MAX_PHYSMEM_BITS, which is also
only complicated on x86 with 5 level page tables.

Luckily the cpu bits are all setup within early_cpu_init(). The setup of
l1tf_mitigation variable is later but still earlier than most of the init
calls (which swapfile_init belongs to level 4).

A full graph for reference:

- start_kernel
  - setup_arch
    - early_cpu_init
      - get_cpu_cap --> fetch from CPUID (including X86_FEATURE_LA57)
      - early_identify_cpu --> setup X86_BUG_L1TF
      - early_identify_cpu --> clear X86_FEATURE_LA57 (if early lvl5 not enabled (USE_EARLY_PGTABLE_L5))
  - parse_early_param
    - l1tf_cmdline --> set l1tf_mitigation
  - check_bugs
    - l1tf_select_mitigation --> set l1tf_mitigation
  - arch_call_rest_init
    - rest_init
      - kernel_init
        - kernel_init_freeable
          - do_basic_setup
            - do_initcalls --> calls swapfile_init() (initcall level 4)

I'll add one (or maybe >1?) patch on top in next post to optimize the
fetching of these states.

Thanks,

-- 
Peter Xu