Re: [PATCH for-next 05/20] RDMA/hns: Add command queue support for hip08 RoCE driver

[Wei Hu (Xavier)]

On 2017/9/27 0:18, Doug Ledford wrote:
> On 9/26/2017 9:13 AM, Wei Hu (Xavier) wrote:
> > On 2017/9/26 1:36, Doug Ledford wrote:
> > > On Mon, 2017-09-25 at 20:18 +0300, Leon Romanovsky wrote:
> > > > On Mon, Sep 25, 2017 at 01:06:53PM -0400, Doug Ledford wrote:
> > > > > On Wed, 2017-08-30 at 17:23 +0800, Wei Hu (Xavier) wrote:
> > > > >
> > > > > > +ÂÂÂ /*
> > > > > > +ÂÂÂÂ * If the command is sync, wait for the firmware to
> > > > > > write
> > > > > > back,
> > > > > > +ÂÂÂÂ * if multi descriptors to be sent, use the first one to
> > > > > > check
> > > > > > +ÂÂÂÂ */
> > > > > > +ÂÂÂ if ((desc->flag) & HNS_ROCE_CMD_FLAG_NO_INTR) {
> > > > > > +ÂÂÂÂÂÂÂ do {
> > > > > > +ÂÂÂÂÂÂÂÂÂÂÂ if (hns_roce_cmq_csq_done(hr_dev))
> > > > > > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ break;
> > > > > > +ÂÂÂÂÂÂÂÂÂÂÂ usleep_range(1000, 2000);
> > > > > > +ÂÂÂÂÂÂÂÂÂÂÂ timeout++;
> > > > > > +ÂÂÂÂÂÂÂ } while (timeout < priv->cmq.tx_timeout);
> > > > > > +ÂÂÂ }
> > > > > then we spin here for a maximum amount of time between 200 and
> > > > > 400ms,
> > > > > so 1/4 to 1/2 a second. All the time we are holding the bh lock on
> > > > > this CPU. That seems excessive to me. If we are going to spin
> > > > > that
> > > > > long, can you find a way to allocate/reserve your resources, send
> > > > > the
> > > > > command, then drop the bh lock while you spin, and retake it before
> > > > > you
> > > > > complete once the spinning is done?
> > > > They don't allocate anything in this loop, but checking the pointers
> > > > are
> > > > the same, see hns_roce_cmq_csq_done.
> > > I'm not sure I understand your intended implication of your comment. I
> > > wasn't concerned about them allocating anything, only that if the
> > > hardware is hung, then this loop will hang out for 1/4 to 1/2 a second
> > > and hold up all bottom half processing on this CPU in the meantime.
> > > That's the sort of things that provides poor overall system behavior.
> > >
> > > Now, since they are really only checking to see if the hardware has
> > > gotten around to their particular command, and their command is part of
> > > a ring structure, it's possible to record the original head command,
> > > and our new head command, and then release the spin_lock_bh around the
> > > entire do{ }while construct, and in hns_roce_cmd_csq_done() you could
> > > check that head is not in the range old_head:new_head. That would
> > > protect you in case something in the bottom half processing queued up
> > > some more commands and from one sleep to the next the head jumped from
> > > something other than the new_head to something past new_head, so that
> > > head == priv->cmq.csq.next_to_use ends up being perpetually false.
> > > But, that's just from a quick read of the code, I could easily be
> > > missing something here...
> > Hi, Doug
> >  ÂÂÂ Driver issues the cmds in cmq, and firmware gets and processes them.
> >  ÂÂÂ The firmware process only one cmd at the same time, and it will take
> >  ÂÂÂ about serveral to 200 us in one cmd currently, so the driver need
> >  ÂÂÂ not to use stream mode to issue cmd.
> I'm not sure I understand your response here.
>
> I get that the driver issues cmds in the cmq, and that the firmware gets
> them and processes them.
>
> I get that the firmware will only work on one command at a time and only
> move to the next one once the current one is complete.
>
> I get that commands take anywhere from a few usec to a couple hundred usec.
>
> I also get that because you are sleeping for somewhere in between 1000
> and 2000 usecs, that the driver could easily finish a whole slew of
> commands.  It could do 10 slow commands, or 100 or more fast commands.
> What this tells me is that the only reason your current implementation
> of hns_roce_cmq_csq_done() works at all is because you keep the device
> locked out from any other commands being put on the queue.  As far as I
> can tell, that's the only way you can guarantee that at some point you
> will wake up and the head pointer will be exactly at csq->next_to_use.
> Otherwise, if you didn't block them out, then you could sleep with the
> head pointer before csq->next_to_use and wake up the next time with it
> already well past csq->next_to_use.  Am I right about that?  While you
> are waiting on this command queue, any other commands are blocked from
> being placed on the command queue?
Hi, Doug,
you are right.
And one "hns_x" ib device only has one command queue in hip08,
other commands will be blocked when waiting on the command queue.
> I don't understand what you mean by "so the driver need not to use
> stream mode to issue cmd".
Sorry, my expression error.
stream -> pipeline

And if you argee, after this patchset has been accepted we will send a 
following up patch :
ÂÂÂ In hns_roce_cmq_send function, replace
ÂÂÂ ÂÂÂ usleep_range(1000, 2000);
ÂÂÂ with the following statement:
ÂÂÂÂÂÂÂÂ udelay(1);
ÂÂÂ And if so, we can avoid using usleep_range function in spin_lock_bh 
spin region,
ÂÂÂ because it probally cause calltrace.

ÂÂÂ Best regards
Wei Hu
> >  ÂÂÂ Regards
> > Wei Hu
> > > > > > +#define HNS_ROCE_CMQ_TX_TIMEOUTÂÂÂÂÂÂÂÂÂÂÂ 200
> > > > > or you could reduce the size of this define...
> > > > >
> > > > > --
> > > > > Doug Ledford <dledford@xxxxxxxxxx>
> > > > >  ÂÂÂÂ GPG KeyID: B826A3330E572FDD
> > > > >  ÂÂÂÂ Key fingerprint = AE6B 1BDA 122B 23B4 265BÂ 1274 B826 A333 0E57
> > > > > 2FDD
> > > > >
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe linux-
> > > > > rdma" in
> > > > > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > > > > More majordomo info at http://vger.kernel.org/majordomo-info.html