Re: [RFC PATCH 0/9] CXL 2.0 Support

[Ben Widawsky]

On 20-12-06 23:40:47, Chris Browy wrote:
> Hi Ben,
> 
> 
> >On Dec 4, 2020, at 1:12 PM, Ben Widawsky <mailto:ben.widawsky@xxxxxxxxx> wrote:
> >
> >Hi Chris.
> >
> >On 20-12-04 12:40:03, Chris Browy wrote:
> >
> >Hi Ben,
> >
> >Trying to bring up the environment using the latest developments as follows:
> >
> >1. Linux kernel baseline version is cloned using
> >     git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
> >   Using master branch.  Merged the 9 CXL linux kernel patches manually and built kernel
> >
> >2. QEMU baseline version is cloned using
> >     git clone https://gitlab.com/bwidawsk/qemu.git
> >
> >3. UEFI baseline is cloned using
> >     git clone https://github.com/tianocore/edk2.git
> >   Using master and built
> >
> >4. Now can run qemu as follows:
> >     The qcow2 we use is based on Ubuntu 20.10 with updated with kernel from 1) above
> >
> >     QEMU command:
> >
> >     sudo qemu-system-x86_64 -nic \
> >     user,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234 -machine \
> >     type=pc-q35-4.0,hmat=on,accel=kvm -enable-kvm -cpu host -smp \
> >     6,cores=6,threads=1,sockets=1 -m 8G -boot order=d -k 'en-us' -vga virtio \
> >     -drive file=/home/chris/Downloads/AQCXL/ubuntu_20.qcow,format=qcow2 -drive \
> >     if=pflash,format=raw,readonly,file=/home/chris/OVMF_CODE.fd \
> >     -drive if=pflash,format=raw,file=/home/chris/OVMF_VARS.fd \
> >     -object memory-backend-file,id=cxl-mem1,share,mem-path=/tmp/cxl-test/cxl,size=512M \
> >     -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,\
> >     window-base[0]=0x4c0000000,memdev[0]=cxl-mem1 \
> >     -device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0  \
> >     -device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M  2>&1 | tee -a \
> >     /home/chris/Downloads/AQCXL/log/qemu.log
> >
> >   The qemu options are derived from looking at the tests/qtests/cxl-test.c
> >   along with the -hmat=on which seemed to make sense.
> >
> >   The system boots and lspci -vvv shows the CXL device is enumerated.  But
> >   no DOE capability register for CDAT access though (see below).  Otherwise the
> >   DVSEC registers are present.
> >
> >DOE is not supported yet in either Linux or QEMU. For us, CDAT isn't a high
> >priority yet so it likely won't be done for a while. I'd really like to see DOE
> >support added by someone - not me - so that we can wire it up. Not sure what
> >that would look like in the QEMU side.
> >
> >
> >
> >   acpidump indicates the CXL0 and CXLM devices but no SRAT or HMAT tables are
> >   in the dump which is curious.
> >
> >I don't typically use HMAT, but I do have an SRAT in mine, so that's strange.
> >You should also have a CEDT.
> >
> Could you provide the QEMU command line?  I was not successful adding numa node for cxl-mem1
> and RAM.  Leaving out numa node for cxl-mem1 I can now see  SRAT table being created but that’s
> not exactly the point.
> 
> Are you using UEFI or legacy BIOS to boot?
> 
> Reproducing a known working environment is better for now and then deviate for trying new
> configurations.

I'm using UEFI. I haven't built my own in a while. The script we use to run QEMU
pulls whatever is here: https://www.archlinux.org/packages/extra/any/edk2-ovmf/download/

-machine q35,accel=kvm,nvdimm,cxl
-m 8192M,slots=4,maxmem=40964M
-smp 8,sockets=2,cores=2,threads=2
-enable-kvm
-display none
-nographic
-drive if=pflash,format=raw,unit=0,file=OVMF_CODE.fd,readonly=on
-drive if=pflash,format=raw,unit=1,file=OVMF_VARS.fd
-debugcon file:uefi_debug.log
-global isa-debugcon.iobase=0x402
-drive file=root.img,format=raw,media=disk
-kernel ./mkosi.extra/boot/vmlinuz
-initrd mkosi.extra/boot/initramfs-5.10.0-rc2-BEN-00016-geb5b35e092af-dirty.img
-append selinux=0 audit=0 console=tty0 console=ttyS0 root=/dev/sda2 ignore_loglevel rw dyndbg="file drivers/cxl/mem.c +p"
-device e1000,netdev=net0,mac=52:54:00:12:34:56
-netdev user,id=net0,hostfwd=tcp::10022-:22
-object memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,window-base[0]=0x4c0000000,memdev[0]=cxl-mem1
-device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0
-device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M
-snapshot
-object memory-backend-ram,id=mem0,size=2048M
-numa node,nodeid=0,memdev=mem0,
-numa cpu,node-id=0,socket-id=0
-object memory-backend-ram,id=mem1,size=2048M
-numa node,nodeid=1,memdev=mem1,
-numa cpu,node-id=1,socket-id=1
-object memory-backend-ram,id=mem2,size=2048M
-numa node,nodeid=2,memdev=mem2,
-object memory-backend-ram,id=mem3,size=2048M
-numa node,nodeid=3,memdev=mem3,
-numa node,nodeid=4,
-object memory-backend-file,id=nvmem0,share,mem-path=nvdimm-0,size=16384M,align=1G
-device nvdimm,memdev=nvmem0,id=nv0,label-size=2M,node=4
-numa node,nodeid=5,
-object memory-backend-file,id=nvmem1,share,mem-path=nvdimm-1,size=16384M,align=1G
-device nvdimm,memdev=nvmem1,id=nv1,label-size=2M,node=5
-numa dist,src=0,dst=0,val=10
-numa dist,src=0,dst=1,val=21
-numa dist,src=0,dst=2,val=12
-numa dist,src=0,dst=3,val=21
-numa dist,src=0,dst=4,val=17
-numa dist,src=0,dst=5,val=28
-numa dist,src=1,dst=1,val=10
-numa dist,src=1,dst=2,val=21
-numa dist,src=1,dst=3,val=12
-numa dist,src=1,dst=4,val=28
-numa dist,src=1,dst=5,val=17
-numa dist,src=2,dst=2,val=10
-numa dist,src=2,dst=3,val=21
-numa dist,src=2,dst=4,val=28
-numa dist,src=2,dst=5,val=28
-numa dist,src=3,dst=3,val=10
-numa dist,src=3,dst=4,val=28
-numa dist,src=3,dst=5,val=28
-numa dist,src=4,dst=4,val=10
-numa dist,src=4,dst=5,val=28
-numa dist,src=5,dst=5,val=10


> 
> >
> >
> >35:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (rev 01) (prog-if 10)
> >    Subsystem: Red Hat, Inc. Device 1100
> >    Physical Slot: 0
> >    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> >    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> >    Latency: 0
> >    Region 0: Memory at c0a00000 (64-bit, non-prefetchable) [size=64K]
> >    Region 2: Memory at c0a10000 (64-bit, non-prefetchable) [size=4K]
> >    Capabilities: [80] Express (v2) Endpoint, MSI 00
> >        DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> >            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
> >        DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
> >            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> >            MaxPayload 128 bytes, MaxReadReq 128 bytes
> >        DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
> >        LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s <64ns
> >            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
> >        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> >            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >        LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
> >            TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
> >        DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR-
> >             10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt+, EETLPPrefix+, MaxEETLPPrefixes 4
> >               EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
> >             FRS-, TPHComp-, ExtTPHComp-
> >             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
> >        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> >             AtomicOpsCtl: ReqEn-
> >        LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
> >             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> >             Compliance De-emphasis: -6dB
> >        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
> >             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> >    Capabilities: [100 v1] Designated Vendor-Specific <?>
> >    Capabilities: [138 v1] Designated Vendor-Specific <?>
> >    Kernel driver in use: cxl_mem
> Can you check this in the QEMU cxl_component_register_init_common() in cxl-component-utils.c
> 
> 
> I used devmem2 to dump out CLX Component CLX.mem regsisters located at BAR + 0x1000. Header
> List:
>      04110000 CXL CAP
>      08010002 CXL RAS
>      0D820004 CXL Link
>      11010005 CXL HDM Decoder
>      00000000
> 
> I think the 1st entry is the CXL Cap Header List should be
> 0x03110001
> 
> 
> Capability_ID looks like it is set using ARRAY_FIELD_DP32
> 
>     /* CXL Capability Header Register */
>     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ID, 1);
>     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, VERSION, 1);
>     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 1);
>     ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps);
> 
> 
> But value reported for ID is ‘0’.
> 
> Since there are only RAS, LINK, and HDM headers found the Array_Size should be ‘3’
> according to the spec:
> 
> Array_Size: This defines the number of elements present in
> the CXL_Capability array, not including the
> CXL_Capability_Header element. Each element is 1 DWORD in
> size and is located contiguous with previous elements.
> 
> 
> ‘caps’ should one less that calculated in switch statement
> 
>  96 void cxl_component_register_init_common(uint32_t *reg_state, enum reg_type type)
>  97 {
>  98     int caps = 0;
>  99     switch (type) {
> 100     case CXL2_DOWNSTREAM_PORT:
> 101     case CXL2_DEVICE:
> 102         /* CAP, RAS, Link */
> 103         caps = 3;
> 104         break;
> 105     case CXL2_UPSTREAM_PORT:
> 106     case CXL2_TYPE3_DEVICE:
> 107     case CXL2_LOGICAL_DEVICE:
> 108         /* + HDM */
> 109         caps = 4;
> 110         break;
> 111     case CXL2_ROOT_PORT:
> 112         /* + Extended Security, + Snoop */
> 113         caps = 6;
> 114         break;
> 115     default:
> 116         abort();
> 117     }
> 
> 

You're correct. I've fixed it and pushed it to the v2 branch.

> >
> >Questions/Comments:
> >-------------------
> >1. Linux
> >  a. Is there a gitlab for the linux kernel patches for CXL?  This would
> >     facilitate review and code modifications.
> >
> >We're hopefully going to send out v2 in the next couple of days. I'll push the
> >repo somewhere as well.
> >
> That’s great!
> 
> >
> >
> >
> >2. UEFI (edk2 from tianocore)
> >  a. seems to only support CXL 1.1 which means only method #1 (Device
> >     option ROM) of Coherent Device Attribute Table_1.02 spec
> >     for CDAT handling is possible now.
> >
> >     Does device option ROM need to be added to QEMU CXL setup?
> >
> >     Can we add a CXL 1.1 emulated device?
> >
> >Patches welcome :-). I know of other people who want this, but I only care about
> >2.0+, so I have no intention to implement it.
> Can you summarize how the System Physical Address (SPA) gets assigned to the
> CXL pmem if not by the UEFI or CXL driver methods?

The PMEM address isn't currently assigned. It expects the driver to program the
HDMs. However, for testing I faked this a bit, grep for SET_PMEM_PADDR. I did
have a tool I used to verify the HDM programming, but it appears that's been
lost to the sands of time.


> 
> Is it some alternate method using host bridge window-base and then
> cxl_realize() in cxl_type3.c.
> 
> -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1, \
>  window-base[0]=0x4c0000000,memdev[0]=cxl-mem1 \
> 
> The QEMU command would inform a lot especially whether you
> create 2 backend memories with numa nodes to models
> RAM and PMEM?
> 
> Eventually the CXL driver would need to handle hot plug of
> CT3D and access CDAT from the CT3D via DOE Mailbox.
> 
> As Dan mentions about Coherent Device Attribute Table (CDAT) Specification r1.02
> requires CDAT to build the SRAT/HMAT statically and dynamically
>  - Figure 1 Pre-boot CDAT Extraction Method (for CXL devices)  (uses CDAT in option ROM)
>  - Figure 2 OS Runtime CDAT Extraction Method (for CXL devices  (uses DOE mailbox)
> 
> >
> >
> >
> >
> >  b. lspci doesn’t show the existence of the DOE extended capability register
> >     in the CXL CT3D (needed to support method #2).  Are there more patches?
> >
> >As above, it's not supported. I'm hoping someone else will do that work since I
> >don't care about it just yet.
> 
> We can get a start on some QEMU updates to support DOE mailbox access.
> 
> >
> >
> >
> >
> >3. Do you have example user programs to share or better yet the CXL 2.0
> >   Sec 14.3.6.1 Application Layer/ Transaction layer test for CXL.mem?
> >
> >I don't have, mostly because I haven't actually implemented a lot of the real
> >CXL support. My primary concern was having the Linux driver be able to enumerate
> >devices and communicate with the device via the mailbox interface. v2 will
> >contain support for userspace to do this, which I think is a step toward what
> >you're asking for.
> >
> Understood. Looking forward to v2 and linux branch!  Working from a known good
> reference environment will allow for more in depth code review.
> 
> 
> >
> >
> >4. What are the userspace system APIs for targeting CXL HDM address domain?
> >   Usually you can mmap a SPA if you know how to look it up.
> >
> >I think Dan answered this in the other thread…
> 
> Correct 
> 
> 
> Best Regards,
> Chris Browy
> 
> 
>