[PATCH] vfio: Request THP-aligned mmap for device fds

[Anthony Pighin]

VFIO PCI devices support PMD-sized page table entries for BAR mappings
via their huge_fault handler (vfio_pci_mmap_huge_fault).  However, the
VFIO device file_operations never provided a get_unmapped_area callback
to request PMD-aligned virtual address placement from the mmap address
allocator.

Before commit 34d7cf637c43 ("mm: don't try THP alignment for FS without
get_unmapped_area"), this was masked by a bug introduced in commit
ed48e87c7df3 ("thp: add thp_get_unmapped_area_vmflags()") which
inadvertently applied THP alignment to all file-backed mappings,
regardless of whether they provided a get_unmapped_area callback.

When commit 34d7cf637c43 ("mm: don't try THP alignment for FS without
get_unmapped_area") correctly restricted THP alignment to anonymous
mappings and files that explicitly opt in via get_unmapped_area, VFIO BAR
mappings lost their PMD-aligned placement.  Since the huge_fault handler
requires both the VMA start address and the physical PFN to be
PMD-aligned, unaligned VMAs force a fallback to 4KB page faults.

For example, a 2GiB BAR results in 524,288 individual page faults
instead of 1,024 PMD-sized faults, increasing the VFIO_IOMMU_MAP_DMA
pinning time by orders of magnitude -- a regression directly visible to
KVM guests during PCI device initialization.

Fix this by providing a get_unmapped_area callback in vfio_device_fops,
following the same pattern used by ext4, xfs, btrfs, fuse, and other
subsystems that benefit from THP-aligned placement.

Fixes: 34d7cf637c43 ("mm: don't try THP alignment for FS without get_unmapped_area")
Cc: stable@xxxxxxxxxxxxxxx
Cc: Alex Williamson <alex@xxxxxxxxxxx>
Cc: Kefeng Wang <wangkefeng.wang@xxxxxxxxxx>
Cc: Vlastimil Babka <vbabka@xxxxxxxxxx>
Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
Cc: kvm@xxxxxxxxxxxxxxx
Signed-off-by: Anthony Pighin <anthony.pighin@xxxxxxxxx>
---
 drivers/vfio/vfio_main.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/vfio/vfio_main.c b/drivers/vfio/vfio_main.c
index 6222376ab6ab..2dbb1a84dbac 100644
--- a/drivers/vfio/vfio_main.c
+++ b/drivers/vfio/vfio_main.c
@@ -40,6 +40,7 @@
 #include <linux/interval_tree.h>
 #include <linux/iova_bitmap.h>
 #include <linux/iommufd.h>
+#include <linux/huge_mm.h>
 #include "vfio.h"
 
 #define DRIVER_VERSION	"0.3"
@@ -1461,6 +1462,7 @@ const struct file_operations vfio_device_fops = {
 	.unlocked_ioctl	= vfio_device_fops_unl_ioctl,
 	.compat_ioctl	= compat_ptr_ioctl,
 	.mmap		= vfio_device_fops_mmap,
+	.get_unmapped_area = thp_get_unmapped_area,
 #ifdef CONFIG_PROC_FS
 	.show_fdinfo	= vfio_device_show_fdinfo,
 #endif
-- 
2.43.0