Re: [RFC PATCH 00/15] io_uring: thread identity handoff for blocking inline issue

[Jens Axboe]

On 9/11/26 11:33 AM, Gabriel Krisman Bertazi wrote:
> Jens Axboe <axboe@xxxxxxxxx> writes:
> 
>> Hi,
>>
>> io_uring issues requests inline with IO_URING_F_NONBLOCK and punts to
>> io-wq when that isn't possible. For a range of opcodes it isn't possible
>> at all, as there's no nonblocking path in the kernel for them: fsync,
>> statx, openat, the *at family, xattr, fadvise, splice, etc. Those are
>> punted unconditionally, and the punt costs a thread wakeup, a context
>> switch and a task_work completion round trip per request. io_uring HAS
>> to be cautious to prevent accidental blocking in the kernel, even if the
>> operations predominantly never block. Sad story. Examples of that are
>> things like an fdatasync that doesn't block, statx that hits dcache,
>> openat for O_TMPFILE, etc. All of those would've completed inline just
>> fine, but io_uring just cannot rely on that.
>>
>> This series issues those requests inline in blocking mode instead, and
>> only pays for the offload if the request actually blocks. But by the
>> time it blocks, the submitter is deep in the kernel with the request on
>> its stack, so the work can't be moved to another thread. What we can
>> move is the identity. If the submitting task blocks, an idle io-wq
>> worker takes over its user visible identity (tid, signal state,
>> credentials, scheduling attributes, cgroup, user register state),
>> finishes the io_uring_enter() call and returns to userspace as the
>> submitter. The original task finishes the request as an
>> io-wq worker and joins the pool. Userspace is none the wiser, hopefully,
>> the same tid came back from the syscall, it's just on a different
>> task_struct. Folks that have been around a while may remember earlier
>> attempts at this about 20 years ago.
> 
> This is both really cool and seems like very dangerous thing :) Count me

Oh yeah, it's definitely crazy and deeply an RFC.

> amazed.  I worry this impersonating method will become as tricky as the
> kthread impersonating model that you replaced with the user workers,
> though.  I haven't looked at your patches yet, but I wonder how you
> handle other tasks that have a reference to your task_struct.

That one was different, because these are normal threads, not kthreads.
They are created similarly to if you did pthread_create() in userspace,
this is what io-wq workers are already. So it's mostly as safe as io-wq
already is, by design, which is why the PF_IO_WORKER work happened and
why kthreads haven't been used since back in the early 5.x days.

So I don't think there's too much to worry about on the security front,
it's mostly a "this will confuse the application" kind of thing because
something has been missed. And yes that is no good either, but it's not
a security concern. That's VERY different from the kthread case, where
if you missed some kind of personality, then congrats you're now running
with fully elevated privileges.

> I was actually working something much simpler to improve this problem,
> which still require subsystems to cooperate, but largely reduces issue:
> 
> My idea was to reuse the non_block_count which already exists in
> task_struct preserved for every kernel config that has io_uring.  We we
> scope the inline path with it.  We then provide new mutex, semaphore
> callers that will check the flag and fail refusing to sleep, similar to
> a try_lock.  The new callers are required because we want subsystems to
> opt-in the behavior, properly clean after themselves, and return
> EWOULDBLOCK.  This is why we need to clean blocking paths in io_uring.
> sched throws a WARN_ON if we schedule out with the counter> 0, making it
> easy to find issues.

I think that would be a tough sell, mostly because of how many locking
primitives we have and how widely they are used, and how difficult (or
impossible) it is to introduce error paths for code that previously had
none. That alone would make it a non-starter for me. Let alone is that
it'd be a continual whack-a-mole kind of work, it'll never be fully
done.

> It has the downside of still requiring fixes to every path and we need
> to handle every new case that comes by, but it is much cleaner than
> plumbing a nonblock flag several layers down the stack across each
> subsystem or having subsystem-specific details in io_uring, which is
> what we have today.  On the upper side, it is much less complex than
> your approach. It also allow us to just back off during memory
> allocations that would block, solving the memory allocations anywhere in
> the submission path, not only inside ->issue(), which we discussed
> recently on discord.

I think you'll find it'll be a lot MORE complicated than my approach!
Backing out error handling is going to be impossible in some cases,
think file systems for example. How would those cases be handled?

> I'll give a try to this series and report back.

Thanks!

-- 
Jens Axboe