Re: [PATCH v2] IB/mlx5: Reduce max order of memory allocated for xlt update
From: Aruna Ramakrishna
Date: Wed Mar 24 2021 - 00:36:14 EST
> On Mar 23, 2021, at 4:13 PM, Jason Gunthorpe <jgg@xxxxxxxx> wrote:
>
> On Tue, Mar 23, 2021 at 12:41:51PM -0700, Aruna Ramakrishna wrote:
>> There is a far greater possibility of an order-8 allocation failing,
>> esp. with the addition of __GFP_NORETRY , and the code would have to
>> fall back to a lower order allocation more often than not (esp. on a
>> long running system). Unless the performance gains from using order-8
>> pages is significant (and it does not seem that way to me), we can just
>> skip this step and directly go to the lower order allocation.
>
> Do not send HTML mails.
I apologize; I’ve fixed the setting now.
>
> Do you have benchmarks that show the performance of the high order
> pages is not relavent? I'm a bit surprised to hear that
>
I guess my point was more to the effect that an order-8 alloc will fail more often than not, in this flow. For instance, when we were debugging the latency spikes here, this was the typical buddyinfo output on that system:
Node 0, zone DMA 0 1 1 2 3 0 1 0 1 1 3
Node 0, zone DMA32 7 7 7 6 10 2 6 7 6 2 306
Node 0, zone Normal 3390 51354 17574 6556 1586 26 2 1 0 0 0
Node 1, zone Normal 11519 23315 23306 9738 73 2 0 1 0 0 0
I think this level of fragmentation is pretty normal on long running systems. Here, in the reg_mr flow, the first try (order-8) alloc will probably fail 9 times out of 10 (esp. after the addition of GFP_NORETRY flag), and then as fallback, the code tries to allocate a lower order, and if that too fails, it allocates a page. I think it makes sense to just avoid trying an order-8 alloc here.
Thanks,
Aruna
> This code really needs some attention to use a proper
> scatter/gather. I understand the chip can do it, just some of the
> software layers need to be stripped away so it can form the right SGL
> in the HW.
>
> Jason