Re: [RFC PATCH] x86, entry: Switch stacks on a paranoid entry from userspace

From: Andy Lutomirski
Date: Tue Nov 18 2014 - 18:04:49 EST

On Tue, Nov 18, 2014 at 10:30 AM, Luck, Tony <tony.luck@xxxxxxxxx> wrote:
>>> The lost cpu is *really* lost. Warm reset doesn't fix the machine, I usually
>>> have to do a full power cycle.
>> How is it even possible that I did that with a few lines of asm?
> Probably not your directly your fault - some cascade of errors may have occurred.

I went and read the manual. Here's a hypothesis:

Your test case is presumably doing something that involves setting
undocumented registers* to program the CPU or memory controller to
generate a machine check on access to some address. Presumably this
is done by broadcasting an SMI and programming the registers in SMM.

Now SMM is rather strange. The docs list a large set of interrupt
sources that are disabled on SMM entry, and this list does not include
#MC. So presumably #MC is actually left enabled on entry to SMM.
That means that, unless SMRAM has an interrupt table that has a
working machine check handler (which seems highly unlikely), then
there is at least some window in which a #MC delivered in SMM will
cause some kind of failure. This could really happen: a broadcast #MC
could easily race a broadcast SMI and do this.

If you crash your SMM code, then I wouldn't be at all surprised if the
CPU wedges hard enough that even your remote management thing can't
reset it.

* These are probably the registers that are supposed to be documented
in volume 2 section 4.4.9 of the Xeon E5 1600/2600 datasheet,
reference 326509-003, but the docs are extremely incomplete.


>> Could this be a hardware bug? Is there some condition that causes #MC
>> delivery to wedge hard enough that even INIT/RESET stops working? Or
>> possibly some CPU got stuck in SMM -- I have no idea what warm reset
>> does these days.
> I'm not even sure what kind of reset the remote management i/f I used
> actually applied.
>> Here's the patch to improve the timeout messages, but given the degree
>> of wedgedness, I can guess what it'll say:
> Heh - I'd already put in some hacky printk()s to do similar. Mine aren't upstream quality, but do print the value of mce_callin/mce_executing
> as appropriate. But I got some confusing results - reporter complained that only 142 of 144 had shown up. So two threads missing,
> maybe means one core went into h/w shutdown. Need to dig further to see if the missing duo really are from the same core.
> -Tony

Andy Lutomirski
AMA Capital Management, LLC
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at