Re: [RFC-PROTOTYPE 1/1] mm: Add __GFP_FAST_TRY flag

[Uladzislau Rezki]

On Tue, Aug 04, 2020 at 07:34:18PM +0200, Vlastimil Babka wrote:
> On 8/4/20 7:12 PM, Matthew Wilcox wrote:
> > On Tue, Aug 04, 2020 at 07:02:14PM +0200, Vlastimil Babka wrote:
> >> > 2) There was a proposal from Matthew Wilcox: https://lkml.org/lkml/2020/7/31/1015
> >> > 
> >> > <snip>
> >> > On non-RT, we could make that lock a raw spinlock.  On RT, we could
> >> > decline to take the lock.  We'd need to abstract the spin_lock() away
> >> > behind zone_lock(zone), but that should be OK.
> >> > <snip>
> >> > 
> >> > It would be great to use any existing flag, say GFP_NOWAIT. Suppose we
> >> > decline to take the lock across the page allocator for RT. But there is
> >> > at least one path that does it outside of the page allocator. GFP_NOWAIT
> >> > can wakeup the kswapd, whereas a "wake-up path" uses sleepable lock:
> >> > 
> >> > wakeup_kswapd() -> wake_up_interruptible(&pgdat->kswapd_wait).
> >> > 
> >> > Probably it can be fixed by the excluding of waking of the kswapd process
> >> > defining something like below:
> >> 
> >> Is something missing here?
> >> 
> >> > what is equal to zero and i am not sure if __get_free_page(0) handles
> >> > all that correctly, though it allocates and seems working on my test
> >> > machine! Please note it is related to "if we can reuse existing flags".
> >> > 
> >> > In the meantime, please see below for a patch that adds a __GFP_FAST_TRY,
> >> > which can at least serve as a baseline against which other proposals can
> >> > be compared. The patch is based on the 5.8.0-rc3.
> >> > 
> >> > Please RFC.
> >> 
> >> At first glance __GFP_FAST_TRY (more descriptive name? __GFP_NO_LOCKS?) seems
> >> better than doing weird things with GFP_NOWAIT, but depends on the real benefits
> >> (hence my first questions).
> > 
> > I think what Vlad is trying to say is that even GFP_NOWAIT will wake
> > kswapd, which involves taking a spinlock.  If you specify 0 in your GFP
> > flags, then we won't wake kswapd.  So a simple:
> > 
> > #define GFP_NOLOCKS	0
> > 
> > should do the trick (modulo various casting, blah blah blah)
> 
> Ah, you're right, waking up kswapd is is only done with __GFP_KSWAPD_RECLAIM and
> GFP_NOWAIT equals to that. So that's easy to avoid for the rcu allocation.
> 
> But still IIUC option 2) would mean that even with "#define GFP_NOLOCKS	0" would
> mean we need to abstract away the zone lock, and behave differently depending on
> the kernel being RT, and inadvertedly changing other users that happen to
> specify gfp where "gfp & GFP_RECLAIM_MASK == 0" (or however we would exactly
> check if we can take the lock on RT kernel). That sounds too complicated to me.
>
I think a different behaviour, i mean RT/non-rt, is not a way forward, because the
things will be over complicated. Please note, the proposed variant is common. It
provides a fast access to pcp-cache, what can be done lock-less. If we could extend
the "fast path" even do the lock-less prefetch(make fast path fully lock-less) from
the body would be fantastic, but that is a bit out of the question.

For example implement removing/inserting pages from "zone->free_area" as lock-less:
llist_add()/llist_del(). But that is theory and on the high level. During investigation
the things might become complicated.

--
Vlad Rezki