Re: [RFC PATCH v1] pin_on_cpu: Introduce thread CPU pinning system call

[Florian Weimer]

* Mathieu Desnoyers:

> It brings an interesting idea to the table though. Let's assume for now that
> the only intended use of pin_on_cpu(2) would be to allow rseq(2) critical
> sections to update per-cpu data on specific cpu number targets. In fact,
> considering that userspace can be preempted at any point, we still need a
> mechanism to guarantee atomicity with respect to other threads running on
> the same runqueue, which rseq(2) provides. Therefore, that assumption does
> not appear too far-fetched.
>
> There are 2 scenarios we need to consider here:
>
> A) pin_on_cpu(2) targets a CPU which is not part of the affinity mask.
>
> This case is easy: pin_on_cpu can return an error, and the caller needs to act
> accordingly (e.g. figure out that this is a design error and report it, or
> decide that it really did not want to touch that per-cpu data that badly and
> make the entire process fall-back to a mechanism which does not use per-cpu
> data at all from that point onwards)

Affinity masks currently are not like process memory: there is an
expectation that they can be altered from outside the process.

Given that the caller may not have any ways to recover from the
suggested pin_on_cpu behavior, that seems problematic.

What I would expect is that if pin_on_cpu cannot achieve implied
exclusion by running on the associated CPU, it acquires a lock that
prevents others pin_on_cpu calls from entering the critical section, and
tasks in the same task group from running on that CPU (if the CPU
becomes available to the task group).  The second part should maintain
exclusion of rseq sequences even if their fast path is not changed.

(On the other hand, I'm worried that per-CPU data structures are a dead
end for user space unless we get containerized affinity masks, so that
contains only see resources that are actually available to them.)

Thanks,
Florian