Re: [RFC PATCH v6 1/5] Thread-local ABI system call: cache CPU number of running thread

[Mathieu Desnoyers]

----- On Apr 4, 2016, at 1:11 PM, H. Peter Anvin hpa@xxxxxxxxx wrote:

> On 04/04/16 10:01, Mathieu Desnoyers wrote:
>> 
>> Changes since v5:
>> - Rename "getcpu_cache" to "thread_local_abi", allowing to extend
>>   this system call to cover future features such as restartable critical
>>   sections. Generalizing this system call ensures that we can add
>>   features similar to the cpu_id field within the same cache-line
>>   without having to track one pointer per feature within the task
>>   struct.
>> - Add a tlabi_nr parameter to the system call, thus allowing to extend
>>   the ABI beyond the initial 64-byte structure by registering structures
>>   with tlabi_nr greater than 0. The initial ABI structure is associated
>>   with tlabi_nr 0.
>> - Rebased on kernel v4.5.
>> 
> 
> This seems absolutely insanely complex, both for the kernel and for
> userspace.
> 
> A much saner way would be for userspace to query the kernel for the size
> of the structure; userspace then allocates the maximum of what it knows
> and what the kernel knows.  That way, the kernel doesn't need to
> conditionalize its accesses to user space, and libc doesn't need to
> conditionalize its accesses either.

If we go down the route of having user-space dynamically allocating
the structure, my understanding is that we need to associate the
user-space TLS symbol with a pointer to the structure, and test for
NULL each time, thus requiring user-space to touch one more cache-line
(read the pointer), and add one conditional per user-space fast-path,
compared to a statically-sized definition approach. Or perhaps you have
some clever trick in mind for "allocation by user-space" that I'm missing ?

Besides the NULL pointer check, another issue is feature detection.
As we extend the feature set, my proposal has a 32-bit features
mask at the beginning of the TLS structure, within the same
cache-line containing the structure fields, so user-space can quickly
check whether the required feature is enabled (adds one conditional
on the user-space fast path, but does not require to touch another
cache-line). This allows adding new features without requiring to
reserve the value "0" within each field of the structure to mean
"feature unavailable", which I find terminally unaesthetic.

I propose here a fixed-size 64 bytes layout for the first structure,
for which a 32-bit feature mask should be enough. If we ever fill
up these 64 bytes, we can then use the following tlabi_nr number (1),
which will define its own structure size and feature mask. This
seems like a good compromise between fast-path speed, feature detection
flexibility, optimal use of cache-lines, and extensibility.

Thanks,

Mathieu


-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com