Re: [LSF/MM] CXL Boot to Bash - Section 4: Interleave
From: Yuquan Wang
Date: Wed Mar 26 2025 - 22:21:32 EST
On Wed, Mar 26, 2025 at 08:53:14AM -0400, Gregory Price wrote:
> On Wed, Mar 26, 2025 at 05:28:00PM +0800, Yuquan Wang wrote:
> > >
> > > Notice at both the root and the host bridge, the Interleave Ways is 2.
> > > There are two targets at each level. The host bridge has a granularity
> > > of 512 to capture its parent's ways and granularity (`2*256`).
> > >
> > > Each decoder is programmed with the total number of targets (4) and the
> > > overall granularity (256B).
> > >
> >
> > Sorry, I tried to set this topology on Qemu Virt and used:
> > "cxl create-region -d decoder0.0 -t ram -m mem0,mem1,mem2,mem3"
> >
> > but it failed with:
> > "cxl region: validate_ways: Interleave ways 2 is less than number of memdevs specified: 4"
> >
> > It seems like the CFMWs IW should be 4?
> >
>
> It has been a while since i've interacted with QEMU's interleave stuff,
> but IIRC (at least back when I was working on it) most configurations
> had 1 device per host bridge - in which case the CFMWS IW should be 4
> with each of the host bridges described in it.
>
> I'm not sure you can do multiple devices per host bridge without a
> switch setup.
>
Qemu counld add 'cxl-rp' under a cxl host bridge. Below is my qemu
command:
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
-device cxl-rp,port=1,bus=cxl.1,id=root_port1,chassis=0,slot=1 \
-device cxl-type3,bus=root_port0,volatile-memdev=mem2,id=cxl-mem1 \
-device cxl-type3,bus=root_port1,volatile-memdev=mem3,id=cxl-mem2 \
-device pxb-cxl,bus_nr=20,bus=pcie.0,id=cxl.2 \
-device cxl-rp,port=2,bus=cxl.2,id=root_port2,chassis=0,slot=2 \
-device cxl-rp,port=3,bus=cxl.2,id=root_port3,chassis=0,slot=3 \
-device cxl-type3,bus=root_port2,volatile-memdev=mem4,id=cxl-mem3 \
-device cxl-type3,bus=root_port3,volatile-memdev=mem5,id=cxl-mem4 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.targets.1=cxl.2,cxl-fmw.0.size=2G \
My lspci shows:
-+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
| +-01.0 Red Hat, Inc. Virtio network device
| +-02.0 Red Hat, Inc. Virtio block device
| +-03.0 Red Hat, Inc. QEMU PCIe Expander bridge
| \-04.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-[0000:0c]-+-00.0-[0d]----00.0 Intel Corporation Device 0d93 (CXL)
| \-01.0-[0e]----00.0 Intel Corporation Device 0d93 (CXL)
\-[0000:14]-+-00.0-[15]----00.0 Intel Corporation Device 0d93 (CXL)
\-01.0-[16]----00.0 Intel Corporation Device 0d93 (CXL)
My cxl list shows:
[
{
"memdev":"mem1",
"ram_size":268435456,
"serial":0,
"host":"0000:15:00.0"
},
{
"memdev":"mem0",
"ram_size":268435456,
"serial":0,
"host":"0000:16:00.0"
},
{
"memdev":"mem2",
"ram_size":268435456,
"serial":0,
"host":"0000:0e:00.0"
},
{
"memdev":"mem3",
"ram_size":268435456,
"serial":0,
"host":"0000:0d:00.0"
}
]
Then:
# cxl create-region -d decoder0.0 -t ram -m mem0,mem1,mem2,mem3
cxl region: validate_ways: Interleave ways 2 is less than number of memdevs specified: 4
cxl region: cmd_create_region: created 0 regions
This case confuesed me :(
Yuquan
> ~Gregory