Re: [PATCH 3/3] objtool/x86: Fix NOP decode

[Alexandre Chartre]

On 9/25/25 14:43, Peter Zijlstra wrote:
> On Thu, Sep 25, 2025 at 12:29:18PM +0100, Andrew Cooper wrote:
> > > I've made that:
> > >
> > > 	case 0x90:
> > > 		if (rex_b) /* XCHG %r8, %rax */
> > > 			break;
> > >
> > > 		if (prefix == 0xf3) /* REP NOP := PAUSE */
> > > 			break;
> > >
> > > 		insn->type = INSN_NOP;
> > > 		break;
> >
> > Legacy prefixes can come in any order.  What is F3 66 90 ?
> >
> > Also, VEX/EVEX/REX2 want excluding too, all of which can encode rex_b
> > differently.
>
> So luckily objtool only really cares about instructions as found in the
> kernel text. Neither f3 66 90 nor VEX/EVEX/REX2 prefixes are of much
> concern.

And it looks like objtool ignores VEX instructions earlier in the same function:

int arch_decode_instruction(struct objtool_file *file, const struct section *sec,
                            unsigned long offset, unsigned int maxlen,
                            struct instruction *insn)
{
       ...

       if (ins.vex_prefix.nbytes)
                return 0;

       ...
}

vex_prefix is set for VEX/EVEX/VEX3/VEX2.

alex.

> But yes.. I happen to have an insn_is_nop() function that can be used on
> userspace, and that certainly wants to be taught about these... x86 is
> such a pain :/
>
> > Is it really only rex_b which prevents NOP becoming a pause, or is it
> > any REX prefix?  I would have thought it was any REX prefix.
>
> SDM opcode table and instruction reference seems consistent with f3
> only.