Re: [PATCH v2 rdma-next] RDMA/mlx5: Enable Relaxed Ordering by default for kernel ULPs

[Max Gurtovoy]

On 6/9/2021 2:05 PM, Leon Romanovsky wrote:
> From: Avihai Horon <avihaih@xxxxxxxxxx>
>
> Relaxed Ordering is a capability that can only benefit users that support
> it. All kernel ULPs should support Relaxed Ordering, as they are designed
> to read data only after observing the CQE and use the DMA API correctly.
>
> Hence, implicitly enable Relaxed Ordering by default for kernel ULPs.
>
> Signed-off-by: Avihai Horon <avihaih@xxxxxxxxxx>
> Signed-off-by: Leon Romanovsky <leonro@xxxxxxxxxx>
> ---
> Changelog:
> v2:
>   * Dropped IB/core patch and set RO implicitly in mlx5 exactly like in
>     eth side of mlx5 driver.
> v1: https://lore.kernel.org/lkml/cover.1621505111.git.leonro@xxxxxxxxxx
>   * Enabled by default RO in IB/core instead of changing all users
> v0: https://lore.kernel.org/lkml/20210405052404.213889-1-leon@xxxxxxxxxx
> ---
>   drivers/infiniband/hw/mlx5/mr.c | 10 ++++++----
>   drivers/infiniband/hw/mlx5/wr.c |  5 ++++-
>   2 files changed, 10 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
> index 3363cde85b14..2182e76ae734 100644
> --- a/drivers/infiniband/hw/mlx5/mr.c
> +++ b/drivers/infiniband/hw/mlx5/mr.c
> @@ -69,6 +69,7 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
>   					  struct ib_pd *pd)
>   {
>   	struct mlx5_ib_dev *dev = to_mdev(pd->device);
> +	bool ro_pci_enabled = pcie_relaxed_ordering_enabled(dev->mdev->pdev);
>   
>   	MLX5_SET(mkc, mkc, a, !!(acc & IB_ACCESS_REMOTE_ATOMIC));
>   	MLX5_SET(mkc, mkc, rw, !!(acc & IB_ACCESS_REMOTE_WRITE));
> @@ -78,10 +79,10 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
>   
>   	if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write))
>   		MLX5_SET(mkc, mkc, relaxed_ordering_write,
> -			 !!(acc & IB_ACCESS_RELAXED_ORDERING));
> +			 acc & IB_ACCESS_RELAXED_ORDERING && ro_pci_enabled);
>   	if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read))
>   		MLX5_SET(mkc, mkc, relaxed_ordering_read,
> -			 !!(acc & IB_ACCESS_RELAXED_ORDERING));
> +			 acc & IB_ACCESS_RELAXED_ORDERING && ro_pci_enabled);

Jason,

If it's still possible to add small change, it will be nice to avoid 
calculating "acc & IB_ACCESS_RELAXED_ORDERING && ro_pci_enabled" twice.