Re: [PATCH] EDAC/{i10nm,skx,skx_common}: Support multiple clumps

[Kyle Meyer]

On Fri, Dec 06, 2024 at 01:26:12AM +0000, Zhuo, Qiuxu wrote:
> > From: Kyle Meyer <kyle.meyer@xxxxxxx>
> > Sent: Friday, December 6, 2024 8:57 AM
> > To: Luck, Tony <tony.luck@xxxxxxxxx>
> > Cc: bp@xxxxxxxxx; james.morse@xxxxxxx; mchehab@xxxxxxxxxx;
> > rric@xxxxxxxxxx; linux-edac@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx
> > Subject: Re: [PATCH] EDAC/{i10nm,skx,skx_common}: Support multiple
> > clumps
> > 
> > On Thu, Dec 05, 2024 at 03:57:55PM -0800, Luck, Tony wrote:
> > > > +int skx_get_src_id(struct skx_dev *d, int off, u8 *id) { #ifdef
> > > > +CONFIG_NUMA
> > > > +	return skx_get_pkg_id(d, id);
> > > > +#else
> > > > +	u32 reg;
> > > > +
> > > > +	if (pci_read_config_dword(d->util_all, off, &reg)) {
> > > > +		skx_printk(KERN_ERR, "Failed to read src id\n");
> > > > +		return -ENODEV;
> > > > +	}
> > > > +
> > > > +	*id = GET_BITFIELD(reg, 12, 14);
> > > > +	return 0;
> > > > +#endif
> > >
> > > Doh ... I alwasy forget about IS_ENABLED(). This can be written:
> > >
> > >
> > > int skx_get_src_id(struct skx_dev *d, int off, u8 *id) {
> > > 	u32 reg;
> > >
> > > 	if (IS_ENABLED(CONFIG_NUMA))
> > > 		return skx_get_pkg_id(d, id);
> > >
> > > 	if (pci_read_config_dword(d->util_all, off, &reg)) {
> > > 		skx_printk(KERN_ERR, "Failed to read src id\n");
> > > 		return -ENODEV;
> > > 	}
> > >
> > > 	*id = GET_BITFIELD(reg, 12, 14);
> > > 	return 0;
> > > }
> > 
> > Looks good.
> > 
> > > 1) Does this work? I tried on a non-clumpy system that is NUMA.
> > 
> > Yes, I just tested this on a Sapphire Rapids system with multiple UPI domains.
> > 
> > > 2) Is it better (assuming #fidef factored off into a .h file)?
> > 
> > IMO, yes, but there's one subtle difference. EDAC will not load on systems that
> > have a single UPI domain when CONFIG_NUMA is enabled but numa=off,
> > because
> > pcibus_to_node() in skx_get_pkg_id() will return NUMA_NO_NODE (-1). Is
> > that a case that we need to worry about?
> 
> I think we need to make the EDAC load/work even in this case. 
> Regardless of CONFIG_NUMA or whether numa=off is set or not, could we do it like this:
> 
> int skx_get_src_id(struct skx_dev *d, int off, u8 *id)
> {
>         u32 reg;
> 
>         if (!skx_get_pkg_id(d, id))
>                 return 0;
> 
>         if (pci_read_config_dword(d->util_all, off, &reg)) {
>                 skx_printk(KERN_ERR, "Failed to read src id\n");
>                 return -ENODEV;
>         }
> 
>         *id = GET_BITFIELD(reg, 12, 14);
>         return 0;
> }

So, we're back to the original issue on systems with multiple UPI/QPI domains
when NUMA is disabled.

Systems with multiple UPI/QPI domains can't use source IDs to map devices
to sockets. skx_get_src_id() will successfully read the source ID from PCI
configuration space registers but it might not map to the correct socket because
each UPI/QPI domain has identical repeating source IDs.

For example, 8 sockets with 2 UPI/QPI domains:

Socket 0 -> Source ID 0
Socket 1 -> Source ID 1
Socket 2 -> Source ID 2
Socket 3 -> Source ID 3
Socket 4 -> Source ID 0
Socket 5 -> Source ID 1
Socket 6 -> Source ID 2
Socket 7 -> Source ID 3

EDAC will successfully load, but it will not find the the corresponding device
for errors on sockets 4 though 7 (for example, see skx_common.c:178).

Looking at my original patch, EDAC will not load when a system has multiple UPI/
QPI domains and NUMA is disabled. We fail early with "Failed to get package ID
from NUMA information" instead of later when an error occurs.

Thanks,
Kyle Meyer