Re: [PATCH v8 12/20] KVM: x86: Support REX2-prefixed opcode decode

[Jim Mattson]

On Wed, Sep 30, 2026 at 3:32 PM Chang S. Bae <chang.seok.bae@xxxxxxxxx> wrote:
>
> Extend the instruction decoder to recognize and handle the REX2 prefix,
> including validation of prefix sequences and correct opcode table
> selection.
>
> REX2 is a terminal prefix: once 0xD5 is encountered, the following byte
> is the opcode. When REX.M=0, most prefix bytes are invalid after REX2,
> including REX, VEX, EVEX, and another REX2. Also, REX2-prefixed
> instructions are only valid in 64-bit mode.
>
> All of the invalid prefix combinations after REX2 coincide with opcodes
> that are architecturally invalid in 64-bit mode. Thus, marking such
> opcodes with No64 in opcode_table[] naturally disallows those illegal
> prefix sequences.

This is only true for 0x40..0x4F, 0xC4, 0xC5, and 0xD5.

Per Section 3.1.2.1 ("REX2 Prefix") of the Intel APX Architecture
Specification (rev. 9.0):

  Furthermore, since the byte following a REX2 prefix is always
  interpreted as the main opcode byte, any legacy prefix byte (namely,
  0x66, 0x67, 0xF0, 0xF2, 0xF3, and segment overrides), a REX prefix
  byte (0x4*), a VEX prefix byte (0xC4 or 0xC5), an EVEX prefix byte
  (0x62), or another REX2 prefix byte (0xD5) following a REX2 prefix
  with REX2.M0 = 0 must #UD, because none of those bytes is the opcode
  of a valid instruction in legacy map 0 in 64-bit mode.

and:

  The 0x0F escape byte is neither needed nor allowed. (That is, the
  REX2 prefix followed by 0x0F triggers #UD.)

In opcode_table[], 0x0F, 0x26, 0x2E, 0x36, 0x3E, 0x62, 0x64, 0x65, 0x66,
0x67, 0xF0, 0xF2, and 0xF3 are all N (NotImpl), not No64. When any of these
bytes follows REX2 with M0=0, KVM will return EMULATION_FAILED to userspace
instead of raising #UD.

> Suggested-by: Paolo Bonzini <pbonzini@xxxxxxxxxx>
> Tested-by: Zhao Liu <zhao1.liu@xxxxxxxxx>
> Signed-off-by: Chang S. Bae <chang.seok.bae@xxxxxxxxx>
> Link: https://lore.kernel.org/CABgObfYYGTvkYpeyqLSr9JgKMDA_STSff2hXBNchLZuKFU+MMA@xxxxxxxxxxxxxx
> ---

[ ... ]

> @@ -4958,6 +4969,17 @@ int x86_decode_insn(struct x86_emulate_ctxt *ctxt, void *insn, int insn_len, int
>                         ctxt->rex_prefix = REX_PREFIX;
>                         ctxt->rex_bits   = ctxt->b & 0xf;
>                         continue;
> +               case 0xd5: /* REX2 */
> +                       if (mode != X86EMUL_MODE_PROT64)
> +                               goto done_prefixes;
> +                       if (rex2_invalid(ctxt)) {
> +                               opcode = ud;
> +                               goto done_modrm;
> +                       }

"ud" is defined as I(SrcNone, emulate_ud), and SrcNone is 0. Jumping to
done_modrm with opcode.flags == 0 sets ctxt->d = 0, which then hits:

        /* Unrecognised? */
        if (ctxt->d == 0)
                return EMULATION_FAILED;

before ctxt->execute is ever set. As a result, rex2_invalid() always
returns EMULATION_FAILED instead of injecting #UD.

Also, because any legacy prefix after a REX prefix resets ctxt->rex_prefix
to REX_NONE at the bottom of the prefix loop, a sequence such as 0x48 0x66
0xD5 will not be caught by rex2_invalid().  Table 3.1 ("Legacy Prefix
Applicability with REX2") lists 0x4* (REX) under "Prefix before REX2" as
"Illegal", whereas the accompanying text says, "A REX prefix (0x4*)
immediately preceding the REX2 prefix is not allowed and triggers #UD."
Does hardware raise #UD on 0x48 0x66 0xD5, or does the intervening legacy
prefix nullify the REX prefix as in legacy encodings?