Keith, while we're on this, regardless of cmb, is SQE memcopy and DB update ordering always guaranteed?
If you look at mlx4 (rdma device driver) that works exactly the same as
nvme you will find:
--
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ qp->sq.head += nreq;
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ /*
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ * Make sure that descriptors are written before
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ * doorbell record.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ */
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ wmb();
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ writel(qp->doorbell_qpn,
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ to_mdev(ibqp->device)->uar_map + MLX4_SEND_DOORBELL);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ /*
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ * Make sure doorbells don't leak out of SQ spinlock
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ * and reach the HCA out of order.
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ */
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ mmiowb();
--