Re: [PATCH v6 2/6] vfio/nvgrace-gpu: Add support for huge pfnmap

[Zhi Wang]

On Tue, 25 Nov 2025 17:30:09 +0000
<ankita@xxxxxxxxxx> wrote:

> From: Ankit Agrawal <ankita@xxxxxxxxxx>
> 
> NVIDIA's Grace based systems have large device memory. The device
> memory is mapped as VM_PFNMAP in the VMM VMA. The nvgrace-gpu
> module could make use of the huge PFNMAP support added in mm [1].
> 
> To make use of the huge pfnmap support, fault/huge_fault ops
> based mapping mechanism needs to be implemented. Currently nvgrace-gpu
> module relies on remap_pfn_range to do the mapping during VM bootup.
> Replace it to instead rely on fault and use vfio_pci_vmf_insert_pfn
> to setup the mapping.
> 
> Moreover to enable huge pfnmap, nvgrace-gpu module is updated by
> adding huge_fault ops implementation. The implementation establishes
> mapping according to the order request. Note that if the PFN or the
> VMA address is unaligned to the order, the mapping fallbacks to
> the PTE level.
> 
> Link:
> https://lore.kernel.org/all/20240826204353.2228736-1-peterx@xxxxxxxxxx/
> [1]
> 
> cc: Shameer Kolothum <skolothumtho@xxxxxxxxxx>
> cc: Alex Williamson <alex@xxxxxxxxxxx>
> cc: Jason Gunthorpe <jgg@xxxxxxxx>
> cc: Vikram Sethi <vsethi@xxxxxxxxxx>
> Signed-off-by: Ankit Agrawal <ankita@xxxxxxxxxx>
> ---
>  drivers/vfio/pci/nvgrace-gpu/main.c | 84
> +++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 22
> deletions(-)
> 
> diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c
> b/drivers/vfio/pci/nvgrace-gpu/main.c index
> e346392b72f6..8a982310b188 100644 ---
> a/drivers/vfio/pci/nvgrace-gpu/main.c +++
> b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -130,6 +130,62 @@ static
> void nvgrace_gpu_close_device(struct vfio_device *core_vdev)
> vfio_pci_core_close_device(core_vdev); }
>  
> +static unsigned long addr_to_pgoff(struct vm_area_struct *vma,
> +				   unsigned long addr)
> +{
> +	u64 pgoff = vma->vm_pgoff &
> +		((1U << (VFIO_PCI_OFFSET_SHIFT - PAGE_SHIFT)) - 1);
> +
> +	return ((addr - vma->vm_start) >> PAGE_SHIFT) + pgoff;
> +}
> +
> +static vm_fault_t nvgrace_gpu_vfio_pci_huge_fault(struct vm_fault
> *vmf,
> +						  unsigned int order)
> +{
> +	struct vm_area_struct *vma = vmf->vma;
> +	struct nvgrace_gpu_pci_core_device *nvdev =
> vma->vm_private_data;
> +	struct vfio_pci_core_device *vdev = &nvdev->core_device;
> +	unsigned int index =
> +		vma->vm_pgoff >> (VFIO_PCI_OFFSET_SHIFT -
> PAGE_SHIFT);
> +	vm_fault_t ret = VM_FAULT_SIGBUS;
> +	struct mem_region *memregion;
> +	unsigned long pfn, addr;
> +
> +	memregion = nvgrace_gpu_memregion(index, nvdev);
> +	if (!memregion)
> +		return ret;
> +
> +	addr = vmf->address & ~((PAGE_SIZE << order) - 1);

ALIGN_DOWN(vmf->address, PAGE_SIZE << order).

> +	pfn = PHYS_PFN(memregion->memphys) + addr_to_pgoff(vma,
> addr); +
> +	if (order && (addr < vma->vm_start ||
> +		      addr + (PAGE_SIZE << order) > vma->vm_end ||
> +		      pfn & ((1 << order) - 1)))

!IS_ALIGNED(pfn, 1 << order).

Other parts looks good to me.

Reviewed-by: Zhi Wang <zhiw@xxxxxxxxxx>