[PATCH -v5] mm, swap: Sort swap entries before free

[Huang Ying]

To reduce the lock contention of swap_info_struct->lock when freeing
swap entry.  The freed swap entries will be collected in a per-CPU
buffer firstly, and be really freed later in batch.  During the batch
freeing, if the consecutive swap entries in the per-CPU buffer belongs
to same swap device, the swap_info_struct->lock needs to be
acquired/released only once, so that the lock contention could be
reduced greatly.  But if there are multiple swap devices, it is
possible that the lock may be unnecessarily released/acquired because
the swap entries belong to the same swap device are non-consecutive in
the per-CPU buffer.

To solve the issue, the per-CPU buffer is sorted according to the swap
device before freeing the swap entries.  Test shows that the time
spent by swapcache_free_entries() could be reduced after the patch.

With the patch, the memory (some swapped out) free time reduced
13.6% (from 2.59s to 2.28s) in the vm-scalability swap-w-rand test
case with 16 processes.  The test is done on a Xeon E5 v3 system.  The
swap device used is a RAM simulated PMEM (persistent memory) device.
To test swapping, the test case creates 16 processes, which allocate
and write to the anonymous pages until the RAM and part of the swap
device is used up, finally the memory (some swapped out) is freed
before exit.

Signed-off-by: Huang Ying <ying.huang@xxxxxxxxx>
Acked-by: Tim Chen <tim.c.chen@xxxxxxxxx>
Cc: Hugh Dickins <hughd@xxxxxxxxxx>
Cc: Shaohua Li <shli@xxxxxxxxxx>
Cc: Minchan Kim <minchan@xxxxxxxxxx>
Cc: Rik van Riel <riel@xxxxxxxxxx>

v5:

- Use a smarter way to determine whether sort is necessary.

v4:

- Avoid unnecessary sort if all entries are from one swap device.

v3:

- Add some comments in code per Rik's suggestion.

v2:

- Avoid sort swap entries if there is only one swap device.
---
 mm/swapfile.c | 43 ++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 38 insertions(+), 5 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 71890061f653..10e75f9e8ac1 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -37,6 +37,7 @@
 #include <linux/swapfile.h>
 #include <linux/export.h>
 #include <linux/swap_slots.h>
+#include <linux/sort.h>
 
 #include <asm/pgtable.h>
 #include <asm/tlbflush.h>
@@ -1065,20 +1066,52 @@ void swapcache_free(swp_entry_t entry)
 	}
 }
 
+static int swp_entry_cmp(const void *ent1, const void *ent2)
+{
+	const swp_entry_t *e1 = ent1, *e2 = ent2;
+
+	return (int)(swp_type(*e1) - swp_type(*e2));
+}
+
 void swapcache_free_entries(swp_entry_t *entries, int n)
 {
 	struct swap_info_struct *p, *prev;
-	int i;
+	int i, m;
+	swp_entry_t entry;
+	unsigned int prev_swp_type;
 
 	if (n <= 0)
 		return;
 
 	prev = NULL;
 	p = NULL;
-	for (i = 0; i < n; ++i) {
-		p = swap_info_get_cont(entries[i], prev);
-		if (p)
-			swap_entry_free(p, entries[i]);
+	m = 0;
+	prev_swp_type = swp_type(entries[0]);
+	for (i = 0; i < n; i++) {
+		entry = entries[i];
+		if (likely(swp_type(entry) == prev_swp_type)) {
+			p = swap_info_get_cont(entry, prev);
+			if (likely(p))
+				swap_entry_free(p, entry);
+			prev = p;
+		} else if (!m)
+			m = i;
+	}
+	if (p)
+		spin_unlock(&p->lock);
+	if (likely(!m))
+		return;
+
+	/* Sort swap entries by swap device, so each lock is only taken once. */
+	sort(entries + m, n - m, sizeof(entries[0]), swp_entry_cmp, NULL);
+	prev = NULL;
+	for (i = m; i < n; i++) {
+		entry = entries[i];
+		if (swp_type(entry) == prev_swp_type)
+			continue;
+		p = swap_info_get_cont(entry, prev);
+		if (likely(p))
+			swap_entry_free(p, entry);
 		prev = p;
 	}
 	if (p)
-- 
2.11.0