Re: [Patch V2] x86, mce: Ensure offline CPU's don't participate in mce rendezvous process.

From: Borislav Petkov
Date: Mon Dec 07 2015 - 15:00:37 EST


On Fri, Dec 04, 2015 at 07:29:36PM -0500, Ashok Raj wrote:
> Linux has logical cpu offline capability. That can be triggered by:
>
> # echo 0 > /sys/devices/system/cpu/cpuX/online
>
> In Intel Architecture, MCE's are broadcasted to all CPUs in the system.
>
> This includes the CPUs marked offline by Linux. Unless the CPU's were removed
> via an ACPI notification, in which case the cpu's are removed from the
> cpu_present_map.
>
> This patch ensures offline CPU's don't participate in MCE rendezvous, but
> simply perform clearing some status bits to ensure a second MCE wont cause
> automatic shutdown.
>
> Without the patch, mce_start will increment mce_callin, but mce_start would
> only wait for all online_cpus. So offline cpu's should avoid participating
> in the rendezvous process.
>
> Reviewed-by: Tony Luck <tony.luck@xxxxxxxxx>
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Ashok Raj <ashok.raj@xxxxxxxxx>
> ---
> arch/x86/kernel/cpu/mcheck/mce.c | 13 +++++++++++++
> 1 file changed, 13 insertions(+)

Tested on a box here, massaged commit message and queued for 4.4,
thanks.

---
From: Ashok Raj <ashok.raj@xxxxxxxxx>
Date: Thu, 3 Dec 2015 19:16:10 -0500
Subject: [PATCH] x86/mce: Ensure offline CPUs don't participate in rendezvous
process

Intel's MCA implementation broadcasts MCEs to all CPUs on the node.
This poses a problem for offlined CPUs which cannot participate in the
rendezvous process:

Kernel panic - not syncing: Timeout: Not all CPUs entered broadcast exception handler
Kernel Offset: disabled
Rebooting in 100 seconds..

More specifically, Linux does a soft offline of a CPU when writing a 0
to /sys/devices/system/cpu/cpuX/online, which doesn't prevent the #MC
exception from being broadcasted to that CPU.

Ensure that offline CPUs don't participate in the MCE rendezvous and
clear the RIP valid status bit so that a second MCE won't cause a
shutdown.

Without the patch, mce_start() will increment mce_callin and wait for
all CPUs. Offlined CPUs should avoid participating in the rendezvous
process altogether.

Reviewed-by: Tony Luck <tony.luck@xxxxxxxxx>
Signed-off-by: Ashok Raj <ashok.raj@xxxxxxxxx>
Cc: "H. Peter Anvin" <hpa@xxxxxxxxx>
Cc: Ingo Molnar <mingo@xxxxxxxxxx>
Cc: linux-edac <linux-edac@xxxxxxxxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx
Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Cc: x86-ml <x86@xxxxxxxxxx>
Link: http://lkml.kernel.org/r/1449188170-3909-1-git-send-email-ashok.raj@xxxxxxxxx
[ Massage commit message. ]
Signed-off-by: Borislav Petkov <bp@xxxxxxx>
---
arch/x86/kernel/cpu/mcheck/mce.c | 11 +++++++++++
1 file changed, 11 insertions(+)

diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 3865e95cc5ec..a006f4cd792b 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -1002,6 +1002,17 @@ void do_machine_check(struct pt_regs *regs, long error_code)
int flags = MF_ACTION_REQUIRED;
int lmce = 0;

+ /* If this CPU is offline, just bail out. */
+ if (cpu_is_offline(smp_processor_id())) {
+ u64 mcgstatus;
+
+ mcgstatus = mce_rdmsrl(MSR_IA32_MCG_STATUS);
+ if (mcgstatus & MCG_STATUS_RIPV) {
+ mce_wrmsrl(MSR_IA32_MCG_STATUS, 0);
+ return;
+ }
+ }
+
ist_enter(regs);

this_cpu_inc(mce_exception_count);
--
2.3.5

--
Regards/Gruss,
Boris.

ECO tip #101: Trim your mails when you reply.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/