RE: [PATCH v2 1/1] scsi: storvsc: Enable scatterlist entry lengths > 4Kbytes

[Michael Kelley]

From: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> Sent: Wednesday, March 3, 2021 1:09 AM
> To: Michael Kelley <mikelley@xxxxxxxxxxxxx>
> Cc: KY Srinivasan <kys@xxxxxxxxxxxxx>; martin.petersen@xxxxxxxxxx; Long Li
> <longli@xxxxxxxxxxxxx>; wei.liu@xxxxxxxxxx; jejb@xxxxxxxxxxxxx; linux-
> hyperv@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; linux-scsi@xxxxxxxxxxxxxxx
> Subject: Re: [PATCH v2 1/1] scsi: storvsc: Enable scatterlist entry lengths > 4Kbytes
> 
> Michael Kelley <mikelley@xxxxxxxxxxxxx> writes:
> 
> > storvsc currently sets .dma_boundary to limit scatterlist entries
> > to 4 Kbytes, which is less efficient with huge pages that offer
> > large chunks of contiguous physical memory. Improve the algorithm
> > for creating the Hyper-V guest physical address PFN array so
> > that scatterlist entries with lengths > 4Kbytes are handled.
> > As a result, remove the .dma_boundary setting.
> >
> > The improved algorithm also adds support for scatterlist
> > entries with offsets >= 4Kbytes, which is supported by many
> > other SCSI low-level drivers.  And it retains support for
> > architectures where possibly PAGE_SIZE != HV_HYP_PAGE_SIZE
> > (such as ARM64).
> >
> > Signed-off-by: Michael Kelley <mikelley@xxxxxxxxxxxxx>
> > ---
> >
> > Changes in v2:
> > * Add HVPFN_DOWN() macro and use it instead of open coding
> >   [Vitaly Kuznetsov]
> > * Change loop that fills pfn array and its initialization
> >   [Vitaly Kuznetsov]
> > * Use offset_in_hvpage() instead of open coding
> >
> >
> >  drivers/scsi/storvsc_drv.c | 66 ++++++++++++++++------------------------------
> >  include/linux/hyperv.h     |  1 +
> >  2 files changed, 24 insertions(+), 43 deletions(-)
> >
> > diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> > index 2e4fa77..5ba3145 100644
> > --- a/drivers/scsi/storvsc_drv.c
> > +++ b/drivers/scsi/storvsc_drv.c
> > @@ -1678,9 +1678,8 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct
> scsi_cmnd *scmnd)
> >  	struct storvsc_cmd_request *cmd_request = scsi_cmd_priv(scmnd);
> >  	int i;
> >  	struct scatterlist *sgl;
> > -	unsigned int sg_count = 0;
> > +	unsigned int sg_count;
> >  	struct vmscsi_request *vm_srb;
> > -	struct scatterlist *cur_sgl;
> >  	struct vmbus_packet_mpb_array  *payload;
> >  	u32 payload_sz;
> >  	u32 length;
> > @@ -1759,8 +1758,8 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct
> scsi_cmnd *scmnd)
> >  	payload_sz = sizeof(cmd_request->mpb);
> >
> >  	if (sg_count) {
> > -		unsigned int hvpgoff = 0;
> > -		unsigned long offset_in_hvpg = sgl->offset & ~HV_HYP_PAGE_MASK;
> > +		unsigned int hvpgoff, hvpfns_to_add;
> > +		unsigned long offset_in_hvpg = offset_in_hvpage(sgl->offset);
> >  		unsigned int hvpg_count = HVPFN_UP(offset_in_hvpg + length);
> >  		u64 hvpfn;
> >
> > @@ -1773,51 +1772,34 @@ static int storvsc_queuecommand(struct Scsi_Host *host,
> struct scsi_cmnd *scmnd)
> >  				return SCSI_MLQUEUE_DEVICE_BUSY;
> >  		}
> >
> > -		/*
> > -		 * sgl is a list of PAGEs, and payload->range.pfn_array
> > -		 * expects the page number in the unit of HV_HYP_PAGE_SIZE (the
> > -		 * page size that Hyper-V uses, so here we need to divide PAGEs
> > -		 * into HV_HYP_PAGE in case that PAGE_SIZE > HV_HYP_PAGE_SIZE.
> > -		 * Besides, payload->range.offset should be the offset in one
> > -		 * HV_HYP_PAGE.
> > -		 */
> >  		payload->range.len = length;
> >  		payload->range.offset = offset_in_hvpg;
> > -		hvpgoff = sgl->offset >> HV_HYP_PAGE_SHIFT;
> >
> > -		cur_sgl = sgl;
> > -		for (i = 0; i < hvpg_count; i++) {
> > +
> > +		for (i = 0; sgl != NULL; sgl = sg_next(sgl)) {
> >  			/*
> > -			 * 'i' is the index of hv pages in the payload and
> > -			 * 'hvpgoff' is the offset (in hv pages) of the first
> > -			 * hv page in the the first page. The relationship
> > -			 * between the sum of 'i' and 'hvpgoff' and the offset
> > -			 * (in hv pages) in a payload page ('hvpgoff_in_page')
> > -			 * is as follow:
> > -			 *
> > -			 * |------------------ PAGE -------------------|
> > -			 * |   NR_HV_HYP_PAGES_IN_PAGE hvpgs in total  |
> > -			 * |hvpg|hvpg| ...              |hvpg|... |hvpg|
> > -			 * ^         ^                                 ^                 ^
> > -			 * +-hvpgoff-+                                 +-hvpgoff_in_page-+
> > -			 *           ^                                                   |
> > -			 *           +--------------------- i ---------------------------+
> > +			 * Init values for the current sgl entry. hvpgoff
> > +			 * and hvpfns_to_add are in units of Hyper-V size
> > +			 * pages. Handling the PAGE_SIZE != HV_HYP_PAGE_SIZE
> > +			 * case also handles values of sgl->offset that are
> > +			 * larger than PAGE_SIZE. Such offsets are handled
> > +			 * even on other than the first sgl entry, provided
> > +			 * they are a multiple of PAGE_SIZE.
> >  			 */
> > -			unsigned int hvpgoff_in_page =
> > -				(i + hvpgoff) % NR_HV_HYP_PAGES_IN_PAGE;
> > +			hvpgoff = HVPFN_DOWN(sgl->offset);
> > +			hvpfn = page_to_hvpfn(sg_page(sgl)) + hvpgoff;
> > +			hvpfns_to_add =	HVPFN_UP(sgl->offset + sgl->length) -
> > +						hvpgoff;
> >
> >  			/*
> > -			 * Two cases that we need to fetch a page:
> > -			 * 1) i == 0, the first step or
> > -			 * 2) hvpgoff_in_page == 0, when we reach the boundary
> > -			 *    of a page.
> > +			 * Fill the next portion of the PFN array with
> > +			 * sequential Hyper-V PFNs for the continguous physical
> > +			 * memory described by the sgl entry. The end of the
> > +			 * last sgl should be reached at the same time that
> > +			 * the PFN array is filled.
> >  			 */
> > -			if (hvpgoff_in_page == 0 || i == 0) {
> > -				hvpfn = page_to_hvpfn(sg_page(cur_sgl));
> > -				cur_sgl = sg_next(cur_sgl);
> > -			}
> > -
> > -			payload->range.pfn_array[i] = hvpfn + hvpgoff_in_page;
> > +			while (hvpfns_to_add--)
> > +				payload->range.pfn_array[i++] =	hvpfn++;
> >  		}
> >  	}
> >
> > @@ -1851,8 +1833,6 @@ static int storvsc_queuecommand(struct Scsi_Host *host, struct
> scsi_cmnd *scmnd)
> >  	.slave_configure =	storvsc_device_configure,
> >  	.cmd_per_lun =		2048,
> >  	.this_id =		-1,
> > -	/* Make sure we dont get a sg segment crosses a page boundary */
> > -	.dma_boundary =		PAGE_SIZE-1,
> >  	/* Ensure there are no gaps in presented sgls */
> >  	.virt_boundary_mask =	PAGE_SIZE-1,
> >  	.no_write_same =	1,
> > diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h
> > index 5ddb479..a1eed76 100644
> > --- a/include/linux/hyperv.h
> > +++ b/include/linux/hyperv.h
> > @@ -1717,6 +1717,7 @@ static inline unsigned long virt_to_hvpfn(void *addr)
> >  #define NR_HV_HYP_PAGES_IN_PAGE	(PAGE_SIZE / HV_HYP_PAGE_SIZE)
> >  #define offset_in_hvpage(ptr)	((unsigned long)(ptr) & ~HV_HYP_PAGE_MASK)
> >  #define HVPFN_UP(x)	(((x) + HV_HYP_PAGE_SIZE-1) >> HV_HYP_PAGE_SHIFT)
> > +#define HVPFN_DOWN(x)	((x) >> HV_HYP_PAGE_SHIFT)
> >  #define page_to_hvpfn(page)	(page_to_pfn(page) *
> NR_HV_HYP_PAGES_IN_PAGE)
> >
> >  #endif /* _HYPERV_H */
> 
> Thank you for implementing my suggestion in v2,
> 
> Reviewed-by:  Vitaly Kuznetsov <vkuznets@xxxxxxxxxx>
> 
> --
> Vitaly

Martin -- Is anything else needed for you to pick up this patch?

Michael