RE: [PATCH 2/2] x86/mce: Add mce=panic_on_ce_count to panic on a corrected error flood

[Luck, Tony]

> > > Why isn't the strategy here page offlining and shutting up the source of the
> > > error instead of doing silly counting and not doing anything to contain the
> > > errors in the first place?
> >
> > While Breno wasn't specific about the source of the errors in these messages, I'm
> > guessing that they might be coming from cache errors rather than DDR memory.
> >
> > There isn't a good way for software[1] to suppress these errors. Taking memory
> > pages offline when the problem is the cache will just deplete available memory
> > without solving the problem.
>
> I have been thinking about this *years* ago. If it is cache errors, we should
> simply offline the core or cores using that cache. We have a lot of cores
> nowadays :)

You aren't the only one. Andi Kleen built this exact idea into mcelog in 2009:

https://git.kernel.org/pub/scm/utils/cpu/mce/mcelog.git/commit/?id=ffd10014622d20eb08fd35b03f756966f80a08bd

This can work well for L1/L2 cache errors. Perhaps also for L3 on modern AMD
CPUs where only small fraction of cores share each L3 cache instance.

L3 on Intel is shared by the whole socket. So you'd lose 50% of cores for an L3 cache
issue on a typical two socket system (plus we'd have to bring back offline of CPU 0
if you want this to work for socket 0).

> In general, us being a lot more resilient and applying automatic containment
> and recovery actions should be the goal IMO.

Taking cores offline likely needs a bunch more plumbing outside of the kernel.
Bare metal systems sometime isolate critical workloads on specific cores. VMM
systems may bind guests to specific cores to provide consistent performance.
Maybe there are already some udev events that could be used to trigger actions
when cores go away?

-Tony