Re: [PATCH v2] cxl/pci: Fix disabling memory if DVSEC CXL Range does not match a CFMWS window

[Robert Richter]

On 16.02.24 20:22:58, Dan Williams wrote:
> Robert Richter wrote:
> > The Linux CXL subsystem is built on the assumption that HPA == SPA.
> > That is, the host physical address (HPA) the HDM decoder registers are
> > programmed with are system physical addresses (SPA).
> > 
> > During HDM decoder setup, the DVSEC CXL range registers (cxl-3.1,
> > 8.1.3.8) are checked if the memory is enabled and the CXL range is in
> > a HPA window that is described in a CFMWS structure of the CXL host
> > bridge (cxl-3.1, 9.18.1.3).
> > 
> > Now, if the HPA is not an SPA, the CXL range does not match a CFMWS
> > window and the CXL memory range will be disabled then. The HDM decoder
> > stops working which causes system memory being disabled and further a
> > system hang during HDM decoder initialization, typically when a CXL
> > enabled kernel boots.
> > 
> > Prevent a system hang and do not disable the HDM decoder if the
> > decoder's CXL range is not found in a CFMWS window.
> > 
> > Note the change only fixes a hardware hang, but does not implement
> > HPA/SPA translation. Support for this can be added in a follow on
> > patch series.
> > 
> > Signed-off-by: Robert Richter <rrichter@xxxxxxx>
> > ---
> >  drivers/cxl/core/pci.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > index a0e7ed5ae25f..18616ca873e5 100644
> > --- a/drivers/cxl/core/pci.c
> > +++ b/drivers/cxl/core/pci.c
> > @@ -478,8 +478,8 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
> >  	}
> >  
> >  	if (!allowed) {
> > -		cxl_set_mem_enable(cxlds, 0);
> > -		info->mem_enabled = 0;
> > +		dev_err(dev, "Range register decodes outside platform defined CXL ranges.\n");
> > +		return -ENXIO;
> 
> While testing I found this needs the following fixup:
> 
> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> index e24ffae8135f..e9e6c81ce034 100644
> --- a/drivers/cxl/core/pci.c
> +++ b/drivers/cxl/core/pci.c
> @@ -477,7 +477,7 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
>                 allowed++;
>         }
>  
> -       if (!allowed) {
> +       if (!allowed && info->mem_enabled) {
>                 dev_err(dev, "Range register decodes outside platform defined CXL ranges.\n");
>                 return -ENXIO;
>         }

The change looks correct to me, thanks for fixing.

-Robert

> 
> 
> ...i.e. Linux should only give up if it does not understand an active
> decode region.
> 
> Now this SPA/HPA mismatch will still cause problems later in region
> creation flow, but that's a separate issue.