Re: [PATCH] perf/x86/intel/uncore: Fix for iio mapping on Skylake Server
From: Meyer, Kyle
Date: Tue Nov 03 2020 - 16:47:46 EST
Hello Alexander,
PCI Uncore mapping details including domains:
On HUB'd based systems each Socket is on it's own Segment, and we use
MMFCG space (0x80000000-0x8FFFFFFF) for segment zero. All the other
segments we use the UV-HUBs ConfigPassthru space (a 64bit config space).
For a 2 chassis, 8 socket system it looks like this: (On a 32 socket
system everything quadruples.)
$ dmsg | grep MMCONFIG
[ 5.128486] PCI: MMCONFIG for domain 0000 [bus 00-fe] at [mem 0x80000000-0x8fefffff] (base 0x80000000)
[ 5.132003] PCI: MMCONFIG for domain 0001 [bus 00-fe] at [mem 0xfff00000000-0xfff0fefffff] (base 0xfff00000000)
[ 5.136003] PCI: MMCONFIG for domain 0002 [bus 00-fe] at [mem 0xfff30000000-0xfff3fefffff] (base 0xfff30000000)
[ 5.140002] PCI: MMCONFIG for domain 0003 [bus 00-fe] at [mem 0xfff20000000-0xfff2fefffff] (base 0xfff20000000)
[ 5.144003] PCI: MMCONFIG for domain 0004 [bus 00-fe] at [mem 0xfff50000000-0xfff5fefffff] (base 0xfff50000000)
[ 5.148003] PCI: MMCONFIG for domain 0005 [bus 00-fe] at [mem 0xfff40000000-0xfff4fefffff] (base 0xfff40000000)
[ 5.152003] PCI: MMCONFIG for domain 0006 [bus 00-fe] at [mem 0xfff70000000-0xfff7fefffff] (base 0xfff70000000)
[ 5.156003] PCI: MMCONFIG for domain 0007 [bus 00-fe] at [mem 0xfff60000000-0xfff6fefffff] (base 0xfff60000000)
[ 5.160207] PCI: MMCONFIG at [mem 0x80000000-0x8fefffff] reserved in E820
[ 5.164000] PCI: MMCONFIG at [mem 0xfff00000000-0xfff0fefffff] reserved in E820
[ 5.172003] PCI: MMCONFIG at [mem 0xfff30000000-0xfff3fefffff] reserved in E820
[ 5.184001] PCI: MMCONFIG at [mem 0xfff20000000-0xfff2fefffff] reserved in E820
[ 5.192001] PCI: MMCONFIG at [mem 0xfff50000000-0xfff5fefffff] reserved in E820
[ 5.200001] PCI: MMCONFIG at [mem 0xfff40000000-0xfff4fefffff] reserved in E820
[ 5.208001] PCI: MMCONFIG at [mem 0xfff70000000-0xfff7fefffff] reserved in E820
[ 5.216002] PCI: MMCONFIG at [mem 0xfff60000000-0xfff6fefffff] reserved in E820
On hubless systems like cooperhawk we put multiple sockets under a segment
(all sockets in a given chassis reside on the same segment). This also
assigns up to 4 sockets per segment depending on how many sockets are
enabled and use a larger MMCFG space (0x80000000-0x9FFFFFFF) thus
# dmesg | grep MMCONFIG
[ 5.760989] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[ 5.763803] PCI: MMCONFIG for domain 0001 [bus 00-ff] at [mem 0x90000000-0x9fffffff] (base 0x90000000)
[ 5.767807] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
[ 5.771805] PCI: MMCONFIG at [mem 0x90000000-0x9fffffff] reserved in E820
Notes:
This last approach is how it is done on a Intel whitebox too. But that
doesn't scale to the number of sockets we support on a HUB'd based system
so we use a different approach.
For Intel to make this flexible, they need to get the PCIe topology
from the ACPI tables. (MCFG/DSDT/SSDT has all the info they need).
Intel does support segments in their default BIOSes starting with
CooperLake so they should also support segments in their OS drivers/tools.
Thanks,
Kyle Meyer
________________________________________
From: Alexander Antonov <alexander.antonov@xxxxxxxxxxxxxxx>
Sent: Tuesday, October 13, 2020 9:37 AM
To: Meyer, Kyle; peterz@xxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; x86@xxxxxxxxxx
Cc: alexander.shishkin@xxxxxxxxxxxxxxx; kan.liang@xxxxxxxxxxxxxxx; alexey.budankov@xxxxxxxxxxxxxxx; ak@xxxxxxxxxxxxxxx; acme@xxxxxxxxxx; mingo@xxxxxxxxxx; Anderson, Russ
Subject: Re: [PATCH] perf/x86/intel/uncore: Fix for iio mapping on Skylake Server
Hello Kyle,
Currently we do not have plans on supporting the Uncore units to IIO PMON
mapping on multiple segment platforms due to a variety of such platforms.
It would be great if you describe your case, I mean how you configure
segments
on your platform. It will help to cover your configuration and determine a
common approach for the mapping algorithm.
Thanks,
Alexander
On 10/09/2020 05:11 PM, Meyer, Kyle wrote:
> Hello Alexander,
>
> Do you plan on supporting multiple segment platforms?
>
> Thanks,
> Kyle Meyer
>
> ________________________________________
> From: alexander.antonov@xxxxxxxxxxxxxxx <alexander.antonov@xxxxxxxxxxxxxxx>
> Sent: Monday, September 28, 2020 5:21 AM
> To: peterz@xxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; x86@xxxxxxxxxx
> Cc: alexander.shishkin@xxxxxxxxxxxxxxx; kan.liang@xxxxxxxxxxxxxxx; alexey.budankov@xxxxxxxxxxxxxxx; ak@xxxxxxxxxxxxxxx; acme@xxxxxxxxxx; mingo@xxxxxxxxxx; alexander.antonov@xxxxxxxxxxxxxxx; Meyer, Kyle; Anderson, Russ
> Subject: [PATCH] perf/x86/intel/uncore: Fix for iio mapping on Skylake Server
>
> From: Alexander Antonov <alexander.antonov@xxxxxxxxxxxxxxx>
>
> Introduced early attributes /sys/devices/uncore_iio_<pmu_idx>/die* are
> initialized by skx_iio_set_mapping(), however, for example, for multiple
> segment platforms skx_iio_get_topology() returns -EPERM before a list of
> attributes in skx_iio_mapping_group will have been initialized.
> As a result the list is being NULL. Thus the warning
> "sysfs: (bin_)attrs not set by subsystem for group: uncore_iio_*/" appears
> and uncore_iio pmus are not available in sysfs. Clear IIO attr_update
> to properly handle the cases when topology information cannot be
> retrieved.
>
> Fixes: bb42b3d39781 ("perf/x86/intel/uncore: Expose an Uncore unit to IIO PMON mapping")
> Reported-by: Kyle Meyer <kyle.meyer@xxxxxxx>
> Suggested-by: Kan Liang <kan.liang@xxxxxxxxxxxxxxx>
> Reviewed-by: Alexei Budankov <alexey.budankov@xxxxxxxxxxxxxxx>
> Reviewed-by: Kan Liang <kan.liang@xxxxxxxxxxxxxxx>
> Signed-off-by: Alexander Antonov <alexander.antonov@xxxxxxxxxxxxxxx>
> ---
> arch/x86/events/intel/uncore_snbep.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c
> index 62e88ad919ff..ccfa1d6b6aa0 100644
> --- a/arch/x86/events/intel/uncore_snbep.c
> +++ b/arch/x86/events/intel/uncore_snbep.c
> @@ -3749,7 +3749,9 @@ static int skx_iio_set_mapping(struct intel_uncore_type *type)
>
> ret = skx_iio_get_topology(type);
> if (ret)
> - return ret;
> + goto clear_attr_update;
> +
> + ret = -ENOMEM;
>
> /* One more for NULL. */
> attrs = kcalloc((uncore_max_dies() + 1), sizeof(*attrs), GFP_KERNEL);
> @@ -3781,8 +3783,9 @@ static int skx_iio_set_mapping(struct intel_uncore_type *type)
> kfree(eas);
> kfree(attrs);
> kfree(type->topology);
> +clear_attr_update:
> type->attr_update = NULL;
> - return -ENOMEM;
> + return ret;
> }
>
> static void skx_iio_cleanup_mapping(struct intel_uncore_type *type)
>
> base-commit: a1b8638ba1320e6684aa98233c15255eb803fac7
> --
> 2.19.1
>