Re: [PATCH 09/10] x86/fpu: Allow restoring signal frames with larger xstate_size

[Andrei Vagin]

On Mon, Jul 6, 2026 at 10:09 AM Chang S. Bae <chang.seok.bae@xxxxxxxxx> wrote:
>
> On 6/30/2026 5:48 PM, Andrei Vagin wrote:
> >
> > In our previous discussion, I explained why the state translation
> > approach doesn't solve the problem: in-flight signal frames on user
> > stacks cannot be translated. When a process is checkpointed while
> > handling a signal, there is an in-flight signal frame residing directly
> > on the thread's user stack. There is no way for userspace tools to
> > reliably discover arbitrary in-flight signal frames embedded in stack
> > memory.
> >
> > We want to support cases where processes using only cluster-wide
> > available features can be migrated from newer to older CPUs. If we
> > migrate from a newer CPU (larger default `user_size`) to an older CPU
> > (smaller `user_size`), enforcing `xstate_size <= fpstate->user_size` in
> > the kernel unconditionally rejects valid signal frames.
> >
> > An fpu translation mechanism already exists in CRIU to restore current
> > per-thread FPU states, making it possible to migrate workloads between
> > different CPUs even today. But there is always a risk that a process is
> > migrated at the wrong moment (while running inside a signal handler).
> > Without this change, failing the size check on that in-flight stack
> > frame can trigger a state corruption.  This patch eliminates that risk.
>
>
> So if I understand correctly, the intended usage model is a cluster of
> machines exposing a compatible set of xfeatures. Migration may occur in
> either direction (old -> new or new -> old), and the process image,
> including any inflight signal frames, is copied verbatim because
> interpreting or translating arbitrary signal frame is either impossible
> or at least not reliable.

You understand it right.

>
> >> With APX, userspace can no longer assume that a higher XSTATE component
> >> number implies a higher offset within the XSAVE image. Going forward,
> >> migration software will likely need a more robust approach that
> >> interprets the layout and transforms the image when moving between
> >> machines with different layouts. With such translation, maybe further
> >> relaxing the kernel-side checker isn't that needed.
> >
> > I think APX was designed to preserve backward compatibility cleanly. And
> > I don't think that we rely on the assumption that a higher XSTATE component
> > number implies a higher offset within the XSAVE image.
> > `xstate_calculate_size()` already finds the topmost feature by offset. The
> > only reason the kernel needs to know the required xstate size is to
> > correctly pre-fault the user memory buffer.
> >
> > While APX reuses the MPX space in the xsave state, it introduces a new
> > feature bit to indicate the presence of its state, which is really what
> > matters. The actual register state that gets restored depends on
> > `task_xfeatures` and the header's `xstate_bv`. Any attempt by XRSTOR to
> > restore a header containing unsupported feature bits in xstate_bv
> > generates a GP fault. This cleanly traps in restore_fpregs_from_user()
> > and fails out, triggering a SIGSEGV.
>
>

I don't think it strictly is required. Even if we mixed MPX and APX
machines in a pool, it would just mean that migrated processes should
not use either feature.

In reality, MPX was deprecated a long time ago, meaning processes
never have the BNDREGS and BNDCSR bits set in their XSTATE
headers. Furthermore, processes using APX states should never be
migrated to non-APX machines.

With this changes, the kernel will correctly reject signals frames with
MPX states on APX machines and vise versa. If a signal frame contains an
MPX state, it will be rejected on an APX machine because XCR0 will lack
the MPX bits, causing XRSTOR to trigger a #GP. The exact same thing
happens if a signal frame containing an APX state is brought to an older
machine. XRSTOR explicitly triggers a #GP exception if a bit in XCR0 is
0 but the corresponding bit in the xstate_bv field of the XSAVE header
is 1. This hardware enforcement occurs regardless of whether the target
bit is present in the requested-feature bitmap.

>
> >
> > I completely understand that some features may be deprecated in the
> > future, but I still believe that for non-deprecated features, component
> > offsets should be fixed across all CPUs within a vendor's family. If
> > this assumption is ever broken, even standard KVM live migration of
> > guest vCPUs would break.
> >
> > Sorry if I missed something. Maybe you can give an example of when
> > this change would work against us?
>
>
> With dynamic XSTATE, suppose a take has opted into a dynamic state and
> therefore carries a larger xstate_size, but it is migrated to another
> machine where the corresponding dynamic permission has not been
> requested. Since, according to the usage model, reliable interpretation
> of arbitrary signal frames is not possible, I assume the migration
> framework does not inspect each individual state to determine whether
> this is still in its init state or not.
>
> If that's the case, is there a possibility that meaningful dynamic state
> could be lost during migration?

Dynamic XSTATE components (like Intel AMX) require a process to
explicitly request permission via arch_prctl(ARCH_REQ_XCOMP_PERM, ...).
During the checkpoint, CRIU detects which dynamic features have been
enabled for each thread and saves this configuration as part of the
process image. During the restore, CRIU invokes arch_prctl() on threads
to request the identical dynamic XSTATE permissions. If the target
machine doesn't support a specific dynamic state, the restore will fail.

>
> More generally, if ever reconsidering the kernel's sanity checking, one
> concern I still have with this change is validation in the presence of
> unknown future feature bits. This series does not run into that problem
> in practice though.

I agree that signal frames containing any unknown xstates should be
rejected.

Thanks,
Andrei