Re: [Question] Core-to-core CAS latency spikes on AMD EPYC 8534P (Siena) cross-CCD

[Sebastian Andrzej Siewior]

On 2026-04-29 19:27:36 [+0300], Ionut Nechita wrote:
> Hi all,
Hi,

> Kernel
> ======
> 
>   Version:      6.12.57-1.stx.136 (PREEMPT_RT)
>   cpuidle:      driver=none (idle uses mwait, C1 only)
>   cpufreq:      acpi-cpufreq, performance governor, boost enabled (~3.1GHz)

not related to _this_ but boost is usually not ideal since changing CPU
frequency requires some time before the CPU gets operational again.
AMD's document
	https://www.amd.com/content/dam/amd/en/documents/epyc-technical-docs/tuning-guides/56263-EPYC-performance-tuning-app-note.pdf

suggest to disable boosting.

> Questions
> =========
> 
> 1. Are multi-us latency spikes on cross-CCD atomic operations a known
>    characteristic of Infinity Fabric under heavy coherency traffic
>    from all CCDs simultaneously?
> 
> 2. For RT workloads requiring cross-CCD communication, is there any
>    kernel-side mitigation (scheduler hints, IF QoS, bandwidth
>    partitioning via MBA/L3 CAT) beyond pinning to a single CCD?

Not sure what you mean here but CAT (or PQoS as AMD calls it) via
resctrl interface might help if you put the task communicating into the
same cache domain.

I do remember that on a Zen 2 (I think) I observed larger latencies when
the (huge) L3 cache was written back to main memory and the CPU blocked.
This was visible since it was in the ms range. You have here ns range
so… The resctl interface mitigated this to some degree. There were news
sites claiming that this kind of latency was solved in a later
generation but I don't find the details now.

I find however https://hothardware.com/news/zen-5-c2c-latency-bug-fixed
where they claim firmware upgrade went from ~180 to ~80 on cross CCD
communication but they don't talk about spikes (also Zen5).

> 3. Could this be related to snoop filter capacity on the IO die
>    (given the 16MB L3 per CCX), or is IF arbitration the more likely
>    bottleneck?
> 
> Not reporting a bug - looking for confirmation that this is expected
> hardware behavior and any guidance for multi-CCD RT deployments on
> EPYC Siena.
> 
> Thanks,
> Ionut Nechita

Sebastian