[RFC] signal: per-thread control over alternate signal stack delivery for selected signals

[Tim Parth]

Hi,

I am looking for guidance on a Linux signal ABI limitation that shows up in multi-runtime processes, specifically a .NET host loading a Go c-shared library.

Disclaimer: I am reporting this from the application/runtime integration side, not as a kernel developer. I arrived here after tracing crashes in a .NET application hosting a Go shared library through several runtime-specific issues, reproductions, and analyses. My understanding of the Linux signal subsystem and ABI details is therefore limited, and I may be missing important details.

The technical summary below reflects my best understanding of the issue based on the referenced investigations. I used AI-assisted editing to help structure and clarify this report, but the observations, reproducer, and referenced analyses come from the linked investigations.

This is not a claim that the current kernel behavior violates the existing ABI. Rather, I believe the current ABI lacks a way for multiple language runtimes in the same process to compose their signal and sigaltstack requirements safely.

Observed failure
================

A .NET process loads a Go shared library built with -buildmode=c-shared and calls it via P/Invoke. Under stress, the process crashes with SIGSEGV while CoreCLR is handling SIGRTMIN for runtime activation / GC suspension.

The reproducer is here:

https://github.com/egonelbre/csharp-go-interop-issue/tree/main/dotnet-go-reproducer

Related runtime issues:

https://github.com/golang/go/issues/78883
https://github.com/dotnet/runtime/issues/127320

The .NET-side analysis shows that the crash happens inside CoreCLR's inject_activation_handler path. The kernel delivered SIGRTMIN on the thread's alternate signal stack, and CoreCLR then ran a call chain deep enough to overflow that stack. In the reported case the per-thread alternate stack installed by CoreCLR was 16 KiB. Increasing it to around 49 KiB avoids the crash in the provided stress test, but that is a runtime-specific mitigation and does not address the general ABI composition problem.

Current ABI interaction
=======================

The problematic interaction is:

1. Signal disposition, including SA_ONSTACK, is per-process.
2. sigaltstack is per-thread.
3. On signal delivery, Linux uses the alternate signal stack if the handler has SA_ONSTACK and the current thread has an alternate stack.
4. The Go runtime documents that non-Go signal handlers must use SA_ONSTACK, because Go may be running on limited stacks. For -buildmode=c-shared, when Go sees an existing signal handler it may turn on SA_ONSTACK and otherwise keep the existing handler.
5. CoreCLR has internal signals such as SIGRTMIN whose handlers may need a different stack policy or a larger stack budget than the alternate stack currently registered on that thread.

The result is that one runtime can make a process-wide SA_ONSTACK decision that affects handlers and threads owned by another runtime. The other runtime can install a larger per-thread sigaltstack, but that becomes an arms race and does not give a runtime any way to express which signals should use which stack policy on a particular thread.

Why existing mechanisms do not fully solve this
===============================================

- Raising SIGSTKSZ or MINSIGSTKSZ does not solve the general issue. The kernel can only know the signal frame requirements, not the maximum user-space stack consumption of an arbitrary signal handler and everything it calls.

- The kernel cannot automatically extend an alternate signal stack.

- Clearing SA_ONSTACK with sigaction is process-wide and can violate the requirements of another runtime, for example Go's requirement that signal handlers run on an alternate stack when Go code may be interrupted.

- SS_AUTODISARM helps with a different class of problems, such as avoiding corruption when switching away from a signal handler, but it does not let a thread express "use an alternate stack for SIGSEGV but not for this runtime-internal suspension signal", nor does it provide separate stack policies for different signals.

Possible ABI direction
======================

One possible direction would be an opt-in, per-thread signal-altstack policy, for example a prctl() or similar interface that lets a thread provide a signal mask for which SA_ONSTACK should be ignored on that thread:
PR_SET_SIGALTSTACK_EXCLUDE_MASK(sigset_t *mask, size_t sigsetsize)

The default mask would be empty, preserving current behavior. Signal delivery would then become, conceptually:

if (handler_has_SA_ONSTACK &&
thread_has_altstack &&
!signal_is_in_current_thread_altstack_exclude_mask)
deliver_on_altstack;
else
deliver_on_normal_stack;

This is only a sketch. I am not attached to this exact interface. Another shape might be preferable, such as a more general per-thread/per-signal alternate stack policy or a way to associate alternate stack requirements with particular signals.

Questions
=========

1. Is the signal maintainers' view that multi-runtime processes should solve this entirely in userspace by agreeing on one sufficiently large per-thread sigaltstack?

2. Would a per-thread/per-signal opt-in policy for alternate signal stack delivery be considered acceptable as a Linux UAPI extension?

3. If such a UAPI is plausible, is prctl() the right place, or would maintainers prefer a different interface?

4. Which subsystem/list should own this discussion? I am sending this first to linux-api and LKML because this appears to be a userspace ABI issue around signal delivery.

Environment from the reproducer report
======================================

- Architecture: x86_64
- OS: Linux
- Example distro: Ubuntu 24.04
- Go: go1.26.2 linux/amd64
- .NET: 10.0.6 and runtime main were tested in the linked report
- Signal involved in the reproducer: SIGRTMIN
- Failure mode: SIGSEGV while running CoreCLR activation handling on the
alternate signal stack

Thanks,

Tim Parth