Re: [PATCH] cacheinfo: Introduce cache id

[Borislav Petkov]

On Fri, Jul 01, 2016 at 07:24:27PM +0000, Yu, Fenghua wrote:
> I haven't tested on AMD. But I think AMD should have the same code.

Again, bear in mind, this is a qemu+kvm guest.

> Could you please check if
> /sys/device/system/cpu/cpu#/cache/index#/shared_cpu_map contains only
> the cpu itself?

Yes it does:

$ grep . -EriIn /sys/devices/system/cpu/cpu*/cache/index*/shared_cpu_map
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_map:1:01
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_map:1:01
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_map:1:01
/sys/devices/system/cpu/cpu1/cache/index0/shared_cpu_map:1:02
/sys/devices/system/cpu/cpu1/cache/index1/shared_cpu_map:1:02
/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map:1:02
/sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_map:1:04
/sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_map:1:04
/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_map:1:04
/sys/devices/system/cpu/cpu3/cache/index0/shared_cpu_map:1:08
/sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_map:1:08
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map:1:08
/sys/devices/system/cpu/cpu4/cache/index0/shared_cpu_map:1:10
/sys/devices/system/cpu/cpu4/cache/index1/shared_cpu_map:1:10
/sys/devices/system/cpu/cpu4/cache/index2/shared_cpu_map:1:10
/sys/devices/system/cpu/cpu5/cache/index0/shared_cpu_map:1:20
/sys/devices/system/cpu/cpu5/cache/index1/shared_cpu_map:1:20
/sys/devices/system/cpu/cpu5/cache/index2/shared_cpu_map:1:20
/sys/devices/system/cpu/cpu6/cache/index0/shared_cpu_map:1:40
/sys/devices/system/cpu/cpu6/cache/index1/shared_cpu_map:1:40
/sys/devices/system/cpu/cpu6/cache/index2/shared_cpu_map:1:40
/sys/devices/system/cpu/cpu7/cache/index0/shared_cpu_map:1:80
/sys/devices/system/cpu/cpu7/cache/index1/shared_cpu_map:1:80
/sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map:1:80

Here's the same from a real AMD system:

 grep . -EriIn /sys/devices/system/cpu/cpu*/cache/index*/shared_cpu_map
/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_map:1:01
/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_map:1:01
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_map:1:01
/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_map:1:3f
/sys/devices/system/cpu/cpu1/cache/index0/shared_cpu_map:1:02
/sys/devices/system/cpu/cpu1/cache/index1/shared_cpu_map:1:02
/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_map:1:02
/sys/devices/system/cpu/cpu1/cache/index3/shared_cpu_map:1:3f
/sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_map:1:04
/sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_map:1:04
/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_map:1:04
/sys/devices/system/cpu/cpu2/cache/index3/shared_cpu_map:1:3f
/sys/devices/system/cpu/cpu3/cache/index0/shared_cpu_map:1:08
/sys/devices/system/cpu/cpu3/cache/index1/shared_cpu_map:1:08
/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map:1:08
/sys/devices/system/cpu/cpu3/cache/index3/shared_cpu_map:1:3f
/sys/devices/system/cpu/cpu4/cache/index0/shared_cpu_map:1:10
/sys/devices/system/cpu/cpu4/cache/index1/shared_cpu_map:1:10
/sys/devices/system/cpu/cpu4/cache/index2/shared_cpu_map:1:10
/sys/devices/system/cpu/cpu4/cache/index3/shared_cpu_map:1:3f
/sys/devices/system/cpu/cpu5/cache/index0/shared_cpu_map:1:20
/sys/devices/system/cpu/cpu5/cache/index1/shared_cpu_map:1:20
/sys/devices/system/cpu/cpu5/cache/index2/shared_cpu_map:1:20
/sys/devices/system/cpu/cpu5/cache/index3/shared_cpu_map:1:3f

L3 is correctly shared between all cores.

> From the cache id you dump on KVM, it tells there are 8 cache leaf 0,
> 8 cache leaf 1, and 8 cache leaf 2. That means each CPU has its own
> 3 caches (I can't tell from the cache id which level they are. The
> cache/index#/level will tell that).
>
> If this is not the case, maybe cache id doesn't work on AMD. Maybe I
> don't enable cache id for AMD?

Nah, leave it as it is. Who knows what we might use it for on AMD.

> From the info, cache id on one level is unique on that level across the board.

Same on the AMD box:

$ grep . -EriIn /sys/devices/system/cpu/cpu*/cache/index*/id
/sys/devices/system/cpu/cpu0/cache/index0/id:1:0
/sys/devices/system/cpu/cpu0/cache/index1/id:1:0
/sys/devices/system/cpu/cpu0/cache/index2/id:1:0
/sys/devices/system/cpu/cpu0/cache/index3/id:1:0
/sys/devices/system/cpu/cpu1/cache/index0/id:1:1
/sys/devices/system/cpu/cpu1/cache/index1/id:1:1
/sys/devices/system/cpu/cpu1/cache/index2/id:1:1
/sys/devices/system/cpu/cpu1/cache/index3/id:1:1
/sys/devices/system/cpu/cpu2/cache/index0/id:1:2
/sys/devices/system/cpu/cpu2/cache/index1/id:1:2
/sys/devices/system/cpu/cpu2/cache/index2/id:1:2
/sys/devices/system/cpu/cpu2/cache/index3/id:1:2
/sys/devices/system/cpu/cpu3/cache/index0/id:1:3
/sys/devices/system/cpu/cpu3/cache/index1/id:1:3
/sys/devices/system/cpu/cpu3/cache/index2/id:1:3
/sys/devices/system/cpu/cpu3/cache/index3/id:1:3
/sys/devices/system/cpu/cpu4/cache/index0/id:1:4
/sys/devices/system/cpu/cpu4/cache/index1/id:1:4
/sys/devices/system/cpu/cpu4/cache/index2/id:1:4
/sys/devices/system/cpu/cpu4/cache/index3/id:1:4
/sys/devices/system/cpu/cpu5/cache/index0/id:1:5
/sys/devices/system/cpu/cpu5/cache/index1/id:1:5
/sys/devices/system/cpu/cpu5/cache/index2/id:1:5
/sys/devices/system/cpu/cpu5/cache/index3/id:1:5

> /sys/devices/system/cpu/cpu0/cache/index0/id:0
> /sys/devices/system/cpu/cpu0/cache/index1/id:0
> /sys/devices/system/cpu/cpu0/cache/index2/id:0
> /sys/devices/system/cpu/cpu0/cache/index3/id:0
> /sys/devices/system/cpu/cpu10/cache/index0/id:17
> /sys/devices/system/cpu/cpu10/cache/index1/id:17
> /sys/devices/system/cpu/cpu10/cache/index2/id:17

Hmm, so non-L3 caches, i.e., the unshared ones, have different IDs from
the CPU numbers. That's probably because we use the APIC IDs to generate
the cache IDs. Not that it matters too much...

...

> /sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,36
> /sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,36
> /sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,36
> /sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-17,36-53

Right, and this shows what is shared by what.

-- 
Regards/Gruss,
    Boris.

SUSE Linux GmbH, GF: Felix ImendÃrffer, Jane Smithard, Graham Norton, HRB 21284 (AG NÃrnberg)
--