[BUG] x86/mce: mce_syscore_resume() queues MCEs with MCP_QUEUE_LOG without draining, causing permanent 20 Hz MCP polling
From: Aditya Kulkarni
Date: Fri Nov 20 2026 - 20:05:22 EST
Hi,
I've run into a bug in arch/x86/kernel/cpu/mce/core.c where resuming
from S3 on a machine whose firmware leaves stale MCA bank status bits
across sleep permanently locks all CPUs into ~20 Hz machine check
polling (~320-335 wakeups/sec across 16 threads) and silently drops the
resume-time MCE logs until a sysfs write to check_interval or reboot.
Environment
-----------
- Hardware: Apple MacBookPro16,1
- CPU: Intel(R) Core(TM) i9-9880H (family 6, model 158, stepping 13,
microcode 0x104, 16 threads)
- Kernel: 7.2.15 (7.2.15-arch1-1-t2; CONFIG_HZ=1000; arch/x86/kernel/cpu/mce/
is unmodified upstream)
- MCE settings: default (check_interval=300, cmci_disabled=0, ignore_ce=0,
mca_cfg.bootlog enabled)
Root cause
----------
1. On this machine, Apple EFI leaves stale error bits in MCA Bank 12 and
Bank 13 (status 0xee2000000040110a, ADDR 0xff7dda00/0xff7dda40) on
both cold boot and S3 resume.
2. At cold boot, __mcheck_cpu_init_prepare_banks() checks mca_cfg.bootlog
and calls:
machine_check_poll(MCP_UC | MCP_QUEUE_LOG, &all_banks);
Because MCP_QUEUE_LOG is set, machine_check_poll() calls
mce_gen_pool_add() without calling irq_work_queue(&mce_irq_work).
Later during boot, mcheck_late_init() checks !mce_gen_pool_empty()
and schedules mce_work, which logs the entries and empties
mce_event_llist.
3. On S3 resume, mce_syscore_resume() calls
__mcheck_cpu_init_prepare_banks() again. Since mca_cfg.bootlog is
still non-zero, it again invokes
machine_check_poll(MCP_UC | MCP_QUEUE_LOG, &all_banks), pushing the
post-S3 Bank 12/13 entries onto mce_event_llist without queuing
mce_irq_work or mce_work.
Unlike mcheck_late_init() and store_int_with_restart() (both of which
check !mce_gen_pool_empty() and schedule mce_work after calling
__mcheck_cpu_init_prepare_banks()), mce_syscore_resume() never drains
mce_event_llist.
4. As a result, mce_event_llist stays permanently non-empty after the
first S3 resume (and those MCEs never appear in dmesg). On every
subsequent mce_timer_fn() tick, !mce_gen_pool_empty() evaluates true
(without scheduling mce_work), so every CPU halves mce_next_interval
down to the HZ/100 (10 jiffies / 10 ms) floor:
- Within ~10 minutes of S3 resume (at check_interval=300), all 16
threads hit the 10 ms floor and stay there until reboot.
- Inside __start_timer(), round_jiffies(jiffies + 10) falls back to
unrounded jiffies + 10 whenever rem <= 249 ms (since rounding down
lands in the past), firing ~20-25 times during the first 250 ms of
every second and then rounding up to the next second boundary.
- Over ~14 hours of active uptime this accumulated ~810k-935k MCP
polls per CPU thread (~320 wakeups/sec system-wide) with THR=0 and
MCE=0.
Live reproducer & verification
------------------------------
1. Fresh boot (22 min uptime, 0 suspends, check_interval=300):
THR: 0 0 0 ...
MCE: 0 0 0 ...
MCP: 5 6 6 ... (1 poll every 300s)
2. Set check_interval=5 (just to speed up the halving sequence from
~10 min to ~15s), suspend to S3 (mem_sleep=deep), and resume ~30s
later (at 19:55:59):
~28s after resume, /proc/interrupts over a 2-second window:
t=0s: MCP: 420 334 325 333 331 327 334 331 324 333 330 337 332 330 337 334
t=2s: MCP: 459 373 364 378 374 370 376 372 364 374 370 382 377 375 380 377
(~20-22.5/sec per thread, ~335/sec total; zero MCE messages in dmesg.)
3. Write 300 to /sys/devices/system/machinecheck/machinecheck0/check_interval
(triggering store_int_with_restart(), which checks
!mce_gen_pool_empty() and queues mce_work):
Immediately at 19:57:00, dmesg flushes the two trapped S3-resume
records (note TIME 1795222559 == 19:55:59, 61 seconds earlier):
Nov 20 19:57:00 macintel kernel: mce: [Hardware Error]: CPU 0:
Machine Check: 0 Bank 12: ee2000000040110a
Nov 20 19:57:00 macintel kernel: mce: [Hardware Error]: TSC 0
ADDR ff7dda00 MISC 43880000086
Nov 20 19:57:00 macintel kernel: mce: [Hardware Error]: PROCESSOR
0:906ed TIME 1795222559 SOCKET 0 APIC 0 microcode 104
Nov 20 19:57:00 macintel kernel: mce: [Hardware Error]: CPU 0:
Machine Check: 0 Bank 13: ee2000000040110a
Nov 20 19:57:00 macintel kernel: mce: [Hardware Error]: TSC 0
ADDR ff7dda40 MISC 43880000086
Nov 20 19:57:00 macintel kernel: mce: [Hardware Error]: PROCESSOR
0:906ed TIME 1795222559 SOCKET 0 APIC 0 microcode 104
And MCP polling immediately drops from ~335/sec back to 0/sec:
t=0s: MCP: 1002 933 920 941 929 917 934 924 905 933 920 941 928
919 933 924
t=2s: MCP: 1002 933 920 941 929 917 934 924 905 933 920 941 928
919 933 924
Workaround & possible fix
-------------------------
Booting with mce=nobootlog avoids the issue by skipping the
machine_check_poll(MCP_UC | MCP_QUEUE_LOG, ...) call in
__mcheck_cpu_init_prepare_banks().
For an upstream fix, mce_syscore_resume() could queue mce_irq_work when
!mce_gen_pool_empty() (e.g. irq_work_queue(&mce_irq_work), matching
mce_log() and safe with IRQs disabled in syscore resume), or
__mcheck_cpu_init_prepare_banks() could avoid MCP_QUEUE_LOG once the
system is past early boot.
Happy to test a patch if you'd like.
Thanks,
Aditya Kulkarni