[PATCH v2 2/2] mm/migrate: raise the do_pages_stat() chunk to 512 pages

[Qiliang Yuan]

do_pages_stat() copies the addresses in and the status out in chunks of
16 on the stack, and do_pages_stat_array() walks each chunk on its own.
With runs of consecutive pages walked in one go, the walk setup paid
every 16 pages now dominates the cost.

Raise the chunk to 512 pages, the number of pages a PTE table maps with
4K pages on x86-64, and allocate the two arrays rather than keeping
6 KiB on the stack. A failed allocation returns -ENOMEM, which
move_pages() already documents.

On 7.3-rc5 in a 16-vCPU VM, querying every page of a populated 4 GiB
buffer, per page and by chunk size:

                16        64        256       512       1024
  4K pages      31.9 ns   15.0 ns   11.8 ns   11.2 ns   10.9 ns
  THP           23.1 ns   6.6 ns    2.5 ns    1.8 ns    1.6 ns

Suggested-by: Zi Yan <ziy@xxxxxxxxxx>
Signed-off-by: Qiliang Yuan <odys.yuan@xxxxxxxxx>
---
 mm/migrate.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index f4d8bfb9b7b4a..9c77b5e5cf1d5 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2656,10 +2656,20 @@ static int do_pages_stat(struct mm_struct *mm, unsigned long nr_pages,
 			 const void __user * __user *pages,
 			 int __user *status)
 {
-#define DO_PAGES_STAT_CHUNK_NR 16UL
-	const void __user *chunk_pages[DO_PAGES_STAT_CHUNK_NR];
-	int chunk_status[DO_PAGES_STAT_CHUNK_NR];
+#define DO_PAGES_STAT_CHUNK_NR 512UL
+	const void __user **chunk_pages;
 	unsigned long chunk_offset = 0;
+	int *chunk_status;
+
+	chunk_pages = kmalloc_array(DO_PAGES_STAT_CHUNK_NR,
+				    sizeof(*chunk_pages), GFP_KERNEL);
+	chunk_status = kmalloc_array(DO_PAGES_STAT_CHUNK_NR,
+				     sizeof(*chunk_status), GFP_KERNEL);
+	if (!chunk_pages || !chunk_status) {
+		kfree(chunk_pages);
+		kfree(chunk_status);
+		return -ENOMEM;
+	}
 
 	while (nr_pages) {
 		unsigned long chunk_nr = min(nr_pages, DO_PAGES_STAT_CHUNK_NR);
@@ -2683,6 +2693,8 @@ static int do_pages_stat(struct mm_struct *mm, unsigned long nr_pages,
 		chunk_offset += chunk_nr;
 		nr_pages -= chunk_nr;
 	}
+	kfree(chunk_pages);
+	kfree(chunk_status);
 	return nr_pages ? -EFAULT : 0;
 }
 

-- 
2.43.0