Re: [RFC PATCH 2/3] vfio-pci: Allow to mmap sub-page MMIO BARs if all MMIO BARs are page aligned

[Alex Williamson]

On Thu, 2015-12-17 at 18:26 +0800, yongji xie wrote:
> 
> On 2015/12/17 4:04, Alex Williamson wrote:
> > On Fri, 2015-12-11 at 16:53 +0800, Yongji Xie wrote:
> > > Current vfio-pci implementation disallows to mmap
> > > sub-page(size < PAGE_SIZE) MMIO BARs because these BARs' mmio
> > > page
> > > may be shared with other BARs.
> > > 
> > > But we should allow to mmap these sub-page MMIO BARs if all MMIO
> > > BARs
> > > are page aligned which leads the BARs' mmio page would not be
> > > shared
> > > with other BARs.
> > > 
> > > This patch adds support for this case and we also add a
> > > VFIO_DEVICE_FLAGS_PCI_PAGE_ALIGNED flag to notify userspace that
> > > platform supports all MMIO BARs to be page aligned.
> > > 
> > > Signed-off-by: Yongji Xie <xyjxie@xxxxxxxxxxxxxxxxxx>
> > > ---
> > > Â drivers/vfio/pci/vfio_pci.cÂÂÂÂÂÂÂÂÂ|ÂÂÂ10 +++++++++-
> > > Â drivers/vfio/pci/vfio_pci_private.h |ÂÂÂÂ5 +++++
> > > Â include/uapi/linux/vfio.hÂÂÂÂÂÂÂÂÂÂÂ|ÂÂÂÂ2 ++
> > > Â 3 files changed, 16 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/vfio/pci/vfio_pci.c
> > > b/drivers/vfio/pci/vfio_pci.c
> > > index 32b88bd..dbcad99 100644
> > > --- a/drivers/vfio/pci/vfio_pci.c
> > > +++ b/drivers/vfio/pci/vfio_pci.c
> > > @@ -443,6 +443,9 @@ static long vfio_pci_ioctl(void *device_data,
> > > ÂÂ		if (vdev->reset_works)
> > > ÂÂ			info.flags |= VFIO_DEVICE_FLAGS_RESET;
> > > Â 
> > > +		if (vfio_pci_bar_page_aligned())
> > > +			info.flags |=
> > > VFIO_DEVICE_FLAGS_PCI_PAGE_ALIGNED;
> > > +
> > > ÂÂ		info.num_regions = VFIO_PCI_NUM_REGIONS;
> > > ÂÂ		info.num_irqs = VFIO_PCI_NUM_IRQS;
> > > Â 
> > > @@ -479,7 +482,8 @@ static long vfio_pci_ioctl(void *device_data,
> > > ÂÂ				ÂÂÂÂÂVFIO_REGION_INFO_FLAG_WRIT
> > > E;
> > > ÂÂ			if (IS_ENABLED(CONFIG_VFIO_PCI_MMAP) &&
> > > ÂÂ			ÂÂÂÂpci_resource_flags(pdev,
> > > info.index) &
> > > -			ÂÂÂÂIORESOURCE_MEM && info.size >=
> > > PAGE_SIZE)
> > > +			ÂÂÂÂIORESOURCE_MEM && (info.size >=
> > > PAGE_SIZE ||
> > > +			ÂÂÂÂvfio_pci_bar_page_aligned()))
> > > ÂÂ				info.flags |=
> > > VFIO_REGION_INFO_FLAG_MMAP;
> > > ÂÂ			break;
> > > ÂÂ		case VFIO_PCI_ROM_REGION_INDEX:
> > > @@ -855,6 +859,10 @@ static int vfio_pci_mmap(void *device_data,
> > > struct vm_area_struct *vma)
> > > ÂÂ		return -EINVAL;
> > > Â 
> > > ÂÂ	phys_len = pci_resource_len(pdev, index);
> > > +
> > > +	if (vfio_pci_bar_page_aligned())
> > > +		phys_len = PAGE_ALIGN(phys_len);
> > > +
> > > ÂÂ	req_len = vma->vm_end - vma->vm_start;
> > > ÂÂ	pgoff = vma->vm_pgoff &
> > > ÂÂ		((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) -
> > > 1);
> > > diff --git a/drivers/vfio/pci/vfio_pci_private.h
> > > b/drivers/vfio/pci/vfio_pci_private.h
> > > index 0e7394f..319352a 100644
> > > --- a/drivers/vfio/pci/vfio_pci_private.h
> > > +++ b/drivers/vfio/pci/vfio_pci_private.h
> > > @@ -69,6 +69,11 @@ struct vfio_pci_device {
> > > Â #define is_irq_none(vdev) (!(is_intx(vdev) || is_msi(vdev) ||
> > > is_msix(vdev)))
> > > Â #define irq_is(vdev, type) (vdev->irq_type == type)
> > > Â 
> > > +static inline bool vfio_pci_bar_page_aligned(void)
> > > +{
> > > +	return IS_ENABLED(CONFIG_PPC64);
> > > +}
> > I really dislike this.ÂÂThis is a problem for any architecture that
> > runs on larger pages, and even an annoyance on 4k hosts.ÂÂWhy are
> > we
> > only solving it for PPC64?
> Yes, I know it's a problem for other architectures. But I'm not sure
> if 
> other archs prefer
> to enforce the alignment of all BARs to be at least PAGE_SIZE which 
> would result in
> some waste of address space.
> 
> So I just propose a prototype and add PPC64 support here. And other 
> archs could decide
> to use it or not by themselves.
> > Can't we do something similar in the core PCI code and detect it?
> So you mean we can do it like this:
> 
> diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
> index d390fc1..f46c04d 100644
> --- a/drivers/pci/pci.h
> +++ b/drivers/pci/pci.h
> @@ -320,6 +320,11 @@ static inline resource_size_t 
> pci_resource_alignment(struct pci_dev *dev,
> ÂÂÂÂÂÂÂÂÂreturn resource_alignment(res);
> Â }
> 
> +static inline bool pci_bar_page_aligned(void)
> +{
> +ÂÂÂÂÂÂÂreturn IS_ENABLED(CONFIG_PPC64);
> +}
> +
> Â void pci_enable_acs(struct pci_dev *dev);
> 
> Â struct pci_dev_reset_methods {
> 
> or add a config option to indicate that PCI MMIO BARs should be page 
> aligned?Â

Yes, I'm thinking of a boot commandline option, maybe one that PPC64
can default to enabled if it chooses to. ÂThe problem is not unique to
PPC64 and the solution should not be unique either. ÂI don't want to
need to revisit this for ARM, which we know is going to be similarly
afflicted. ÂThanks,

Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/