Re: [PATCH v5] net/mlx5: Reclaim max 50K pages at once

[Anand Khoje]

On 6/28/24 21:14, David Laight wrote:
> ...
> > The way Mellanox ConnectX5 driver handles 'release of allocated pages
> > from HCA' or 'allocation of pages to HCA', is by sending an event to the
> > host. This event will have number of pages in it. If the number is
> > positive, that indicates HCA is requesting that number of pages to be
> > allocated. And if that number is negative, it is the HCA indicating that
> > that number of pages can be reclaimed by the host.
> A one line comment would do.
> Possibly even negating the be32toh() result?
>
> > In this patch we are restricting the maximum number of pages that can be
> > reclaimed to be 50000 (effectively this would be -50000 as it is
> > reclaim). This limit is based on the capability of the firmware as it
> > cannot release more than 50000 back to the host in one go.
> Hang on, why are you soft limiting it to the hard limit?
> I thought the problem was that releasing a lot of pages took a long
> time and 'stuffed' other time-critical tasks.
>
> The only way to resolve that would seem to be to defer the actual freeing
> to a low (or at least normal user) priority thread.
> You would definitely want to get out of 'softint' context.
> (Which is out of napi unless forced to be threaded - and that only really
> works if you force the threads under the RT scheduler.)
>
> 	David

Hi David,

The issue here is, when Mellanox device sends a huge number of pages 
back to the host to reclaim, the host allocates a certain number of 
mailbox messages mlx5_cmd_mailbox to accommodate the DMA addresses of 
the memory to be reclaimed. The freeing of these mailbox messages is 
time consuming (not the freeing of actual pages).

Now, the limit of the FW is that presently, it frees upto 50000 pages. 
This limit can increase in future firmware versions. We are limiting 
this in the driver because we see optimal results with this limit during 
our tests. The results indicated that the time consumed while freeing of 
mailbox messages stayed 2 usec on average - which is tolerable and would 
not need running this thread in a different (low priority) context.

Thanks,

Anand

> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)