Re: [Question] Should direct reclaim time be bounded?
From: Mike Kravetz
Date: Tue Apr 23 2019 - 12:40:27 EST
On 4/23/19 12:19 AM, Michal Hocko wrote:
> On Mon 22-04-19 21:07:28, Mike Kravetz wrote:
>> In our distro kernel, I am thinking about making allocations try "less hard"
>> on nodes where we start to see failures. less hard == NORETRY/NORECLAIM.
>> I was going to try something like this on an upstream kernel when I noticed
>> that it seems like direct reclaim may never end/exit. It 'may' exit, but I
>> instrumented __alloc_pages_slowpath() and saw it take well over an hour
>> before I 'tricked' it into exiting.
>>
>> [ 5916.248341] hpage_slow_alloc: jiffies 5295742 tries 2 node 0 success
>> [ 5916.249271] reclaim 5295741 compact 1
>
> This is unexpected though. What does tries mean? Number of reclaim
> attempts? If yes could you enable tracing to see what takes so long in
> the reclaim path?
tries is the number of times we pass the 'retry:' label in
__alloc_pages_slowpath. In this specific case, I am pretty sure all that
time is in one call to __alloc_pages_direct_reclaim. My 'trick' to make this
succeed was to "echo 0 > nr_hugepages" in another shell.
>> This is where it stalled after "echo 4096 > nr_hugepages" on a little VM
>> with 8GB total memory.
>>
>> I have not started looking at the direct reclaim code to see exactly where
>> we may be stuck, or trying really hard. My question is, "Is this expected
>> or should direct reclaim be somewhat bounded?" With __alloc_pages_slowpath
>> getting 'stuck' in direct reclaim, the documented behavior for huge page
>> allocation is not going to happen.
>
> Well, our "how hard to try for hugetlb pages" is quite arbitrary. We
> used to rety as long as at least order worth of pages have been
> reclaimed but that didn't make any sense since the lumpy reclaim was
> gone.
Yes, that is what I am seeing in our older distro kernel and I can at least
deal with that.
> So the semantic has change to reclaim&compact as long as there is
> some progress. From what I understad above it seems that you are not
> thrashing and calling reclaim again and again but rather one reclaim
> round takes ages.
Correct
> That being said, I do not think __GFP_RETRY_MAYFAIL is wrong here. It
> looks like there is something wrong in the reclaim going on.
Ok, I will start digging into that. Just wanted to make sure before I got
into it too deep.
BTW - This is very easy to reproduce. Just try to allocate more huge pages
than will fit into memory. I see this 'reclaim taking forever' behavior on
v5.1-rc5-mmotm-2019-04-19-14-53. Looks like it was there in v5.0 as well.
--
Mike Kravetz