Re: [RFC PATCH] resource: Fix CXL node not populated issue

[Dan Williams]

Bharata B Rao wrote:
> On 11-Dec-24 10:14 AM, Dan Williams wrote:
> > 
> > I was able to reproduce a similar set of conditions with the cxl_test
> > environment:
> > 
> > f010000000-f04fffffff : Soft Reserved
> >    f010000000-f04fffffff : CXL Window 0
> >      f020000000-f03fffffff : region3
> >        f020000000-f03fffffff : dax3.0
> >          f020000000-f03fffffff : System RAM (kmem)
> > 
> > 
> > ...but that did not result in the bug. So there are some other details
> > missing. Can you proceed with providing the dmesg from the good and the
> > bad cases?
> > 
> > gist.github.com is useful for this.
> 
> Sorry for the delay in providing the data. The system wasn't available 
> for a while. Now I have put the good(6.11.0-rc6) and bad(6.13.0-rc1) 
> dmesg and iomem logs at
> 
> https://gist.github.com/bharata/4a57db11e044fd1d313035f3dd5f763b
> 
> What I see is that in the bad case, we hit the below shown dev_dbg 
> message (not seen in the logs that are provided)
> 
> static int hmem_register_device(struct device *host, int target_nid,
>                                  const struct resource *res)
> {
>          struct platform_device *pdev;
>          struct memregion_info info;
>          long id;
>          int rc;
> 
>          if (IS_ENABLED(CONFIG_CXL_REGION) &&
>              region_intersects(res->start, resource_size(res), 
> IORESOURCE_MEM,
>                                IORES_DESC_CXL) != REGION_DISJOINT) {
>                  dev_dbg(host, "deferring range to CXL: %pr\n", res); <--
>                  return 0;
>          }
> 
> With this, it appears that dev_dax_kmem_probe() won't happen for the CXL 
> range and hence CXL memory doesn't get detected.
> 
> Let us know if you need more debug data.

Ah, thanks for that!

So, it turns out Ying's patch is working as advertised. It is traversing
past the top-level entry of the the iomem_resource topology to discover that a
Soft Reserved range is described by CXL. Then the expectation is that
the CXL subsystem parses the topology and registers a dax device.

I missed that detail in my repro because I was not testing the HMEM
handoff.

Now, the dmesg shows that the CXL subsystem gives up early on the
CXL devices as they do not appear to be meeting the expecations of a "CXL
Memory Device" as described by the "PCI Header - Class Code Register
(Offset 09h)" of the CXL spec.

Specifically these messages:

    [    4.449072] cxl_pci 0000:9f:00.0: registers not found: status mbox memdev 

I think this situation is increasingly showing that the pace of
non-standard quirks being deployed is higher than CXL subsystem is able
to keep pace.

The immediate workaround to this problem that Linux discovered is to
disable the cxl_acpi driver. Can you confirm that preventing the
cxl_acpi driver from loading restores that missing node?

Longer term the urgency of Nathan's patch [1] needs to be escalated.

[1]: http://lore.kernel.org/20241202155542.22111-1-nathan.fontenot@xxxxxxx