Re: Restartable Sequences system call merged into Linux

From: Florian Weimer
Date: Wed Jun 13 2018 - 04:21:42 EST

On 06/12/2018 06:31 PM, Mathieu Desnoyers wrote:
----- On Jun 12, 2018, at 9:11 AM, Florian Weimer fweimer@xxxxxxxxxx wrote:

On 06/11/2018 10:04 PM, Mathieu Desnoyers wrote:
----- On Jun 11, 2018, at 3:55 PM, Florian Weimer fweimer@xxxxxxxxxx wrote:

On 06/11/2018 09:49 PM, Mathieu Desnoyers wrote:
It should be noted that there can be only one rseq TLS area registered per
which can then be used by many libraries and by the executable, so this is a
process-wide (per-thread) resource that we need to manage carefully.

Is it possible to resize the area after thread creation, perhaps even
from other threads?

I'm not sure why we would want to resize it. The per-thread area is fixed-size.
Its layout is here: include/uapi/linux/rseq.h: struct rseq

Looks I was mistaken and this is very similar to the robust mutex list.

Should we treat it the same way? Always allocate it for each new thread
and register it with the kernel?

That would be an efficient way to do it, indeed. There is very little
performance overhead to have rseq registered for all threads, whether or
not they intend to run rseq critical sections.

The ABI is designed so that all users (program and libraries) can interact
through this per-thread TLS area.

Then the user code needs just the address of the structure.


So we'd add

struct rseq *rseq_location (void);

and be done with it? It would return the address of the thread-local variable, similar to __errno_location.

Or we could add something like this:

extern __thread struct rseq pthread_rseq_area_np
__attribute__ ((__tls_model__ ("initial-exec")));

But of course only for recent-enough GNU compilers (and Clang, which identifies itself as GNU).

The advantage of the function call is that it often results in more compact code. Making the initial-exec nature part of the ABI has the advantage that the applications could use the fact of the constant offset to the thread pointer if they desire to do so.

Would we need to document which glibc functions use pthread_rseq_area_np, so that applications do not call them when they itself use the area?

Do we actually need to use RSEQ_FLAG_UNREGISTER prior to thread exit? Why can't the kernel do it for us?

- requires all rseq users to upgrade to newer glibc. Early rseq users
(libs and applications) registering their own rseq TLS will conflict
with newer glibc.

We will need to do something about stack unwinding and longjmp anyway (I assume the kernel already handles signals for us), so it may not be possible to use restartable sequences in any substantial way with a system upgrade anyway.

B) exposes a strong __rseq_abi symbol:

- should ideally *not* be global-dynamic for performance reasons, but
testing shows that using initial-exec causes issues in situations where ends up being dlopen'd (e.g. java virtual machine dlopening
the lttng-ust tracer linked against,

Just an aside:

You can work around that using preloading. On the glibc side, we could also make the initial reserve configurable. On 64-bit, there really is no reason not to use a different TCB allocation scheme which would allow you to create a few threads before the initial-exec TLS area cannot be extended.

The existing approach dates back to LinuxThreads and its TCB collocated with the the stack. But changes in the next few months are not very likely.

C) __rseq_abi symbol declared weak within each user (application, librseq,
other libraries, glibc):

We can multiple two non-weak definitions for the symbol. It should work as long as only the definition in glibc has a symbol version.

__rseq_abi as a name is problematic because it's in the internal namespace.