Re: [PATCH] RDMA/mlx4: Spread completion vectors for proxy CQs

[HÃkon Bugge]

> On 20 Feb 2019, at 18:14, Jason Gunthorpe <jgg@xxxxxxxx> wrote:
> 
> On Tue, Feb 19, 2019 at 06:32:50PM +0100, HÃkon Bugge wrote:
>>   Anyway, Jason mentioned in a private email that maybe we could use the
>>   new completion API or something? I am not familiar with that one
>>   (yet).
> 
> I was thinking of the stuff in core/cq.c - but it also doesn't have
> automatic comp_vector balancing. It is the logical place to put
> something like that though..
> 
> An API to manage a bundle of CPU affine CQ's is probably what most
> ULPs really need.. (it makes little sense to create a unique CQ for
> every QP)

ULPs behave way differently. E.g. RDS creates one tx and one rx CQ per QP.

As I wrote earlier, we do not have any modify_cq() that changes the comp_vector (EQ association). We can balance #CQ associated with the EQs, but we do not know their behaviour.

So, assume 2 completion EQs, and four CQs. CQa and CQb are associated with the first EQ, the two others with the second EQ. That's the "best" we can do. But, if CQa and CQb are the only ones generating events, we will have all interrupt processing on a single CPU. But if we now could modify CQa.comp_vector to be that of the second EQ, we could achieve balance. But not sure if the drivers are able to do this at all.

> alloc_bundle()

You mean alloc a bunch of CQs? How do you know their #cqes and cq_context?


HÃkon


> get_cqn_for_flow(bundle)
> alloc_qp()
> destroy_qp()
> put_cqn_for_flow(bundle)
> destroy_bundle();
> 
> Let the core code balance the cqn's and allocate (shared) CQ
> resources.
> 
> Jason