Re: AMD EPYC 25 (19h): Hardware Error: Machine Check: 0 Bank 17: d42040000000011b

[Borislav Petkov]

On Wed, Apr 12, 2023 at 05:11:26PM +0200, Paul Menzel wrote:
> On a Dell PowerEdge R7525 with AMD EPYC 7763 64-Core Processor, Linux
> 5.15.94 logs the machine check exceptions (MCE) below:
> 
> ```
> [5154053.127240] mce: [Hardware Error]: Machine check events logged
> [5154053.133711] mce: [Hardware Error]: CPU 3: Machine Check: 0 Bank 17:
> d42040000000011b
> [5154053.141948] mce: [Hardware Error]: TSC 0 ADDR b3cbdbbc0 PPIN
> 2b615bef7f48098 SYND 6bd210000a801002 IPID 9600650f00

Build the latest kernel with CONFIG_X86_MCE_INJECT and
CONFIG_EDAC_DECODE_MCE enabled and CONFIG_RAS_CEC *disabled*. Then boot
it on that machine with and do the following below.

The files are in debugfs:

/sys/kernel/debug/mce-inject/
├── addr
├── bank
├── cpu
├── flags
├── ipid
├── misc
├── README
├── status
└── synd

so you go and do

echo 0xd42040000000011b > status
echo 0xb3cbdbbc0 > addr
echo 3 > cpu
echo "sw" > flags
echo 0x6bd210000a801002 > synd
echo 0x9600650f00 > ipid
echo 17 > bank

Remember to keep the bank write last because this one injects the error.

It should dump the decoded error in dmesg.

Alternatively, if you have CONFIG_EDAC_DECODE_MCE enabled on the
machine and you boot with "ras=cec_disable", it would decode it
automatically so you don't have to do it yourself.

Below's the full help text how to do the injection.

And yeah, I know, this is not a very user-friendly way how to decode
those but we're working on one...

HTH.

static const char readme_msg[] =
"Description of the files and their usages:\n"
"\n"
"Note1: i refers to the bank number below.\n"
"Note2: See respective BKDGs for the exact bit definitions of the files below\n"
"as they mirror the hardware registers.\n"
"\n"
"status:\t Set MCi_STATUS: the bits in that MSR control the error type and\n"
"\t attributes of the error which caused the MCE.\n"
"\n"
"misc:\t Set MCi_MISC: provide auxiliary info about the error. It is mostly\n"
"\t used for error thresholding purposes and its validity is indicated by\n"
"\t MCi_STATUS[MiscV].\n"
"\n"
"synd:\t Set MCi_SYND: provide syndrome info about the error. Only valid on\n"
"\t Scalable MCA systems, and its validity is indicated by MCi_STATUS[SyndV].\n"
"\n"
"addr:\t Error address value to be written to MCi_ADDR. Log address information\n"
"\t associated with the error.\n"
"\n"
"cpu:\t The CPU to inject the error on.\n"
"\n"
"bank:\t Specify the bank you want to inject the error into: the number of\n"
"\t banks in a processor varies and is family/model-specific, therefore, the\n"
"\t supplied value is sanity-checked. Setting the bank value also triggers the\n"
"\t injection.\n"
"\n"
"flags:\t Injection type to be performed. Writing to this file will trigger a\n"
"\t real machine check, an APIC interrupt or invoke the error decoder routines\n"
"\t for AMD processors.\n"
"\n"
"\t Allowed error injection types:\n"
"\t  - \"sw\": Software error injection. Decode error to a human-readable \n"
"\t    format only. Safe to use.\n"
"\t  - \"hw\": Hardware error injection. Causes the #MC exception handler to \n"
"\t    handle the error. Be warned: might cause system panic if MCi_STATUS[PCC] \n"
"\t    is set. Therefore, consider setting (debugfs_mountpoint)/mce/fake_panic \n"
"\t    before injecting.\n"
"\t  - \"df\": Trigger APIC interrupt for Deferred error. Causes deferred \n"
"\t    error APIC interrupt handler to handle the error if the feature is \n"
"\t    is present in hardware. \n"
"\t  - \"th\": Trigger APIC interrupt for Threshold errors. Causes threshold \n"
"\t    APIC interrupt handler to handle the error. \n"
"\n"
"ipid:\t IPID (AMD-specific)\n"
"\n";

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette