[PATCH 1/7] mm/userfaultfd: Make uffd read() wait event exclusive

[Peter Xu]

From: Andrea Arcangeli <aarcange@xxxxxxxxxx>

When a new message is generated for an userfaultfd, instead of waking up
all the readers, we can wake up only one exclusive reader to process the
event.  Waking up >1 readers for 1 message will be a waste of resource,
where the rest readers will see nothing again and re-queue.

This should make userfaultfd read() O(1) on wakeups.

Note that queuing on head is intended (rather than tail) to make sure the
readers are waked up in LIFO fashion; fairness doesn't matter much here,
but caching does.

Signed-off-by: Andrea Arcangeli <aarcange@xxxxxxxxxx>
[peterx: modified subjects / commit message]
Signed-off-by: Peter Xu <peterx@xxxxxxxxxx>
---
 fs/userfaultfd.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 56eaae9dac1a..f7fda7d0c994 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -1061,7 +1061,11 @@ static ssize_t userfaultfd_ctx_read(struct userfaultfd_ctx *ctx, int no_wait,
 
 	/* always take the fd_wqh lock before the fault_pending_wqh lock */
 	spin_lock_irq(&ctx->fd_wqh.lock);
-	__add_wait_queue(&ctx->fd_wqh, &wait);
+	/*
+	 * Only wake up one exclusive reader each time there's an event.
+	 * Paired with wake_up_poll() when e.g. a new page fault msg generated.
+	 */
+	__add_wait_queue_exclusive(&ctx->fd_wqh, &wait);
 	for (;;) {
 		set_current_state(TASK_INTERRUPTIBLE);
 		spin_lock(&ctx->fault_pending_wqh.lock);
-- 
2.41.0