RE: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table update with the lock

[Geethasowjanya Akula]

>-----Original Message-----
>From: Leon Romanovsky <leon@xxxxxxxxxx> 
>Sent: Thursday, March 30, 2023 12:46 PM
>To: Geethasowjanya Akula <gakula@xxxxxxxxxxx>
>Cc: Sai Krishna Gajula <saikrishnag@xxxxxxxxxxx>; davem@xxxxxxxxxxxxx; edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx; pabeni@xxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Sunil >Kovvuri Goutham <sgoutham@xxxxxxxxxxx>; richardcochran@xxxxxxxxx
>Subject: Re: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table update with the lock
>
>On Thu, Mar 30, 2023 at 06:56:54AM +0000, Geethasowjanya Akula wrote:
>> 
>> >-----Original Message-----
>> >From: Leon Romanovsky <leon@xxxxxxxxxx>
>> >Sent: Thursday, March 30, 2023 11:26 AM
>> >To: Sai Krishna Gajula <saikrishnag@xxxxxxxxxxx>
>> >Cc: davem@xxxxxxxxxxxxx; edumazet@xxxxxxxxxx; kuba@xxxxxxxxxx; 
>> >pabeni@xxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; 
>> >linux-kernel@xxxxxxxxxxxxxxx; Sunil Kovvuri Goutham 
>> ><sgoutham@xxxxxxxxxxx>; >richardcochran@xxxxxxxxx; Geethasowjanya 
>> >Akula <gakula@xxxxxxxxxxx>
>> >Subject: [EXT] Re: [net PATCH 1/7] octeontx2-af: Secure APR table 
>> >update with the lock
>> 
>> >External Email
>> 
>> >---------------------------------------------------------------------
>> >- On Wed, Mar 29, 2023 at 10:36:13PM +0530, Sai Krishna wrote:
>> >> From: Geetha sowjanya <gakula@xxxxxxxxxxx>
>> >> 
>> >> APR table contains the lmtst base address of PF/VFs.
>> >> These entries are updated by the PF/VF during the device probe. Due 
>> >> to race condition while updating the entries are getting corrupted. 
>> >> Hence secure the APR table update with the lock.
>> 
>> >However, I don't see rsrc_lock in probe path.
>> >otx2_probe()
>> >-> cn10k_lmtst_init()
>> > -> lmt_base/lmstst is updated with and without mbox.lock.
>> 
>> >Where did you take rsrc_lock in probe flow?
>> 
>> rsrc_lock is initialized in AF driver. PF/VF driver in cn10k_lmtst_init() send a mbox request to AF to update the lmtst table. 
>> mbox handler in AF takes rsrc_lock to update lmtst table.

>Can you please present the stack trace of such flow? What are the actual variables/struct rsrc_lock is protecting?

The lock tries to protect the request and response register at line#73 and line#83 in below function, from getting overwritten when
Multiple PFs invokes rvu_get_lmtaddr() simultaneously. 
For example, if PF1 submit the request at line#73 and got permitted before it reads the response at line#80.
PF2 got scheduled submit the request then the response of PF1 is overwritten by the PF2 response.  
When PF1 gets reschedule, it reads wrong data.

#static int rvu_get_lmtaddr(struct rvu *rvu, u16 pcifunc,
  59                            u64 iova, u64 *lmt_addr)
  60 {
  61        [...]
  68 
  69         rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_SMMU_ADDR_REQ, iova);
  70         pf = rvu_get_pf(pcifunc) & 0x1F;
  71         val = BIT_ULL(63) | BIT_ULL(14) | BIT_ULL(13) | pf << 8 |
  72               ((pcifunc & RVU_PFVF_FUNC_MASK) & 0xFF);
  73         rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_SMMU_TXN_REQ, val);
  74 
  75         err = rvu_poll_reg(rvu, BLKADDR_RVUM, RVU_AF_SMMU_ADDR_RSP_STS, BIT_ULL(0), false);
  76         if (err) {
  77                 dev_err(rvu->dev, "%s LMTLINE iova transulation failed\n", __func__);
  78                 return err;
  79         }
  80         val = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_SMMU_ADDR_RSP_STS);
  81         if (val & ~0x1ULL) {
  82                 dev_err(rvu->dev, "%s LMTLINE iova transulation failed err:%llx\n", __func__, val);
  83                 return -EIO;
  84         }
  85         
  
Thanks.
>>Thanks

>> 
>> Thanks,
>> Geetha.
>> 
>> >Thanks
>> 
>> >> 
>> >> Fixes: 893ae97214c3 ("octeontx2-af: cn10k: Support configurable 
>> >> LMTST
>> >> regions")
>> >> Signed-off-by: Geetha sowjanya <gakula@xxxxxxxxxxx>
>> >> Signed-off-by: Sunil Kovvuri Goutham <sgoutham@xxxxxxxxxxx>
>> >> Signed-off-by: Sai Krishna <saikrishnag@xxxxxxxxxxx>
>> >> ---
>> >>  drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c | 8 +++++---
>> >>  1 file changed, 5 insertions(+), 3 deletions(-)
>> >> 
>> >> diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c
>> >> b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c
>> >> index 4ad9ff025c96..8530250f6fba 100644
>> >> --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c
>> >> +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cn10k.c
>> >> @@ -142,16 +142,17 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu,
>> >>  	 * region, if so, convert that IOVA to physical address and
>> >>  	 * populate LMT table with that address
>> >>  	 */
>> >> +	mutex_lock(&rvu->rsrc_lock);
>> >>  	if (req->use_local_lmt_region) {
>> >>  		err = rvu_get_lmtaddr(rvu, req->hdr.pcifunc,
>> >>  				      req->lmt_iova, &lmt_addr);
>> >>  		if (err < 0)
>> >> -			return err;
>> >> +			goto error;
>> >>  
>> >>  		/* Update the lmt addr for this PFFUNC in the LMT table */
>> >>  		err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, lmt_addr);
>> >>  		if (err)
>> >> -			return err;
>> >> +			goto error;
>> >>  	}
>> >>  
>> >>  	/* Reconfiguring lmtst map table in lmt region shared mode i.e. 
>> >> make @@ -181,7 +182,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu,
>> >>  		 */
>> >>  		err = rvu_update_lmtaddr(rvu, req->hdr.pcifunc, val);
>> >>  		if (err)
>> >> -			return err;
>> >> +			goto error;
>> >>  	}
>> >>  
>> >>  	/* This mailbox can also be used to update word1 of 
>> >> APR_LMT_MAP_ENTRY_S @@ -230,6 +231,7 @@ int rvu_mbox_handler_lmtst_tbl_setup(struct rvu *rvu,
>> >>  	}
>> >>  
>> >>  error:
>> >> +	mutex_unlock(&rvu->rsrc_lock);
>> >>  	return err;
>> >>  }
>> >>  
>> >> --
>> >> 2.25.1
>> >>