Re: [PATCH] x86/microcode: Add an option to reload microcode even if revision is unchanged

[Johannes Erdfelt]

On Fri, Sep 06, 2019, Borislav Petkov <bp@xxxxxxxxx> wrote:
> On Fri, Sep 06, 2019 at 07:40:39AM -0700, Johannes Erdfelt wrote:
> > I ask because we have successfully used late microcode loading on tens
> > of thousands of hosts.
> 
> How do you deal with all the mitigations microcode loaded late?

We developed livepatches to add the necessary support. I understand we
aren't the typical Linux user. We do custom development and validation
to support our use case.

That said, we very much rely on late microcode loading and it has helped
us and our customers significantly.

> > I'm a bit worried to see that there is a push to remove a feature that
> > we currently rely on.
> 
> I'd love to remove it. And the fact that people rely on it more instead
> of fixing their infrastructure to reboot machines and do early microcode
> updates is making it worse. Microcode update should be batched with
> kernel updates and that's it. They happen normally once-twice per year -
> except the last two years but the last two years are not normal anyway
> - and done. No need to do some crazy CPUID features reloading dances in
> the kernel and making sure cores will see the updated paths and so on.

It's really easy to say "fix your infrastructure" when you're not
running that infrastructure.

Reboots suck. Customers hate it. Operations hates it. When you get into
the number of hosts we have, you run into all kinds of weird failure
scenarios. (What do you mean that the NIC that was working just fine
before the reboot is no longer seen on the PCI bus?)

The more reboots we can avoid, the better it is for us and our
customers.

I understand that it could be unsafe to late load some rare microcode
updates (theoretical or not). However, that is certainly the exception.
We have done this multiple times on our fleet and we plan to continue
doing so in the future.

There just aren't any good alternatives currently.

JE