Re: userfaultfd: two-step UFFDIO_API always gives -EINVAL

From: Peter Xu
Date: Tue Nov 26 2024 - 12:41:36 EST


On Tue, Nov 26, 2024 at 07:16:19PM +0300, stsp wrote:
> 26.11.2024 18:56, Peter Xu пишет:
> > This doesn't sound like the right thing to do.. as the fd (returned from
> > syscall(userfaultfd)) should be linked to a specific mm. If the parent
> > invoked that syscall, it's linked to the parent address space, not child.
> > You may want to do syscall(userfalut) in child process, then pass it over
> > with scm rights. Otherwise IIUC the trap will be armed on parent virtual
> > address space.
> Ok, thanks for info.
> man page doesn't seem to describe
> the multi-process case, so both fork()
> and SCM_RIGHTS were just a guesses
> on my side, one of which worked.
> Probably something to add to the doc.
>
> The last problem I had (last one, I promise! :)
> is that if I remove O_NONBLOCK, then
> the entire app hangs. It turns out, w/o
> O_NONBLOCK, userfaultfd's fd awakes
> the select() call with the ready-to-read
> descriptor at the very beginning, long

I highly suspect it's not a real POLLIN, but POLLERR. See:

userfaultfd_poll():
/*
* poll() never guarantees that read won't block.
* userfaults can be waken before they're read().
*/
if (unlikely(!(file->f_flags & O_NONBLOCK)))
return EPOLLERR;

I suppose select() will report that in readfds[].

> before any fault is detected. Then it
> goes to read() and blocks forever. My
> code is not prepared for read() blocking
> after select().
> I then checked and double-checked
> and re-checked that with O_NONBLOCK
> nothing like that happens at all: select()
> is not awaken until the faults are coming.
> It could be that select awakes anyway
> but read() doesn block, but no, its not
> the case. In nonblock mode select()
> awakes only when it should. And in
> blocking mode - it awakes immediately,
> leading to a hang.
> Is this a bug?

Not a bug, but, AFAIU, a design decision. If you're interested, you can
read commit ba85c702e4b.

Userfaultfd is a special kind of fd, and poll()/select() doesn't always
mean that the next read() is not going to block. Fundamentally it's
because data-ready event is based on waitqueue, while waitqueue can change
between a select() v.s. a read() later, so the waited entry can be removed
within the short period.

In short, please stick with NONBLOCK on userfaultfd.

Thanks,

--
Peter Xu