Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices

From: Dave Jiang

Date: Mon Nov 16 2026 - 14:11:54 EST




On 11/5/26 11:58 AM, Srirangan Madhavan wrote:
> Hi folks,
>
> This series adds CXL Reset for Type 2 devices through the existing PCI
> reset_method and reset sysfs attributes. It disables CXL.cache, performs
> supported cache writeback, initiates CXL Reset with Memory Clear, and
> restores cached HDM decoder and CXL Device DVSEC state before PCI use
> resumes. Active HDM ranges are reserved and CPU caches invalidated before
> and after reset. The method remains limited to eligible single-function
> devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
> the same cached state and flushes the host and device caches around the
> reset.
>
> The reset sequence follows the CXL 4.0 specification [4], including
> CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
> ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
> define the completion timeout encodings and reset access restrictions.
>
> The userspace ABI is the existing PCI reset interface:
>
> /sys/bus/pci/devices/.../reset_method
> /sys/bus/pci/devices/.../reset
>
> Userspace selects "cxl_reset" in reset_method and triggers the reset
> through reset. Following Dan's v6 feedback, this uses the PCI interface
> instead of adding a memdev-specific sysfs ABI.
>
> For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
> callback when provided and disables PCI I/O, memory decoding, and bus
> mastering. The CXL path reserves affected HDM ranges exclusively and
> blocks IOMMU translations where applicable. It does not offline memory
> or revoke mappings: userspace is expected to offline CXL.mem users before
> reset. The reservation enforces this for claimed ranges by failing the
> reset while a conflicting resource is claimed [9].
>
> The global region lock is held from snapshot through restoration. Memory
> Clear completion is awaited for the timeout the device advertises, up to
> 100 seconds, so the lock can be held for that long plus cache flushes.

I think holding it across the bus reset can deadlock with pciehp and a
PCI remove. I tested in QEMU with two cxl-type2 devices from Manish's
RFC [1].

For a device in a pciehp slot, pci_reset_bus_function() ends up in
pciehp_reset_slot(), which takes the slot's reset_lock. The reset still
holds cxl_rwsem.region at that point.

Two other tasks can complete a cycle. pciehp_ist() for that slot can
hold reset_lock and wait for pci_rescan_remove_lock in
pciehp_unconfigure_device(). A PCI remove of a second Type 2 device can
hold pci_rescan_remove_lock while cxl_decoder_detach() waits for
cxl_rwsem.region.

On v17 as posted, lockdep reports the cycle:

[ 60.891789] WARNING: possible circular locking dependency detected
[ 60.893014] irq/24-pciehp/103 is trying to acquire lock:
[ 60.893367] ffff888072b4f128 (&cxl_memdev_key){+.+.}-{4:4}, at: device_del+0x8d/0x580
but task is already holding lock:
[ 60.894498] ffffffff84699e80 (pci_rescan_remove_lock){+.+.}-{4:4}, at: pciehp_unconfigure_device+0x93/0x210
[...]
-> #3 (&ctrl->reset_lock){++++}-{4:4}:
[ 60.916794] down_write_nested+0x80/0x170
[ 60.917826] pciehp_reset_slot+0x6c/0xc0
[ 60.918893] pci_reset_hotplug_slot+0x5c/0x90
[ 60.919928] pci_reset_bus_function+0x161/0x330
[ 60.920951] cxl_reset_bus_function+0x1c6/0x280
[ 60.921958] __pci_reset_function_locked+0x96/0x170
[ 60.922990] pci_reset_function+0xc8/0x180
[ 60.923977] reset_store+0x84/0xd0
[...]
-> #2 (cxl_rwsem.region){++++}-{4:4}:
[...]
[ 61.013490] Chain exists of:
&cxl_memdev_key --> &ctrl->reset_lock --> pci_rescan_remove_lock

With a test-only msleep() after cxl_region_quiesce_lock() in
pci_cxl_reset_prepare(), it turns into a real hang. The reset, a sysfs
remove of the second device and a power off of the first device's slot
all got stuck.

Two region reads blocked behind the reset as well. A read of
region1/commit timed out after 5 s, and cxl list -R stayed in D state in
decoders_committed_show().

To reproduce, I used Manish's RFC as posted. It allows one cxl-type2 per
pxb-cxl, so each device has its own host bridge and window. On q35 with
cxl=on, plus the memory backends:

-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=53
-device cxl-rp,id=hb0rp0,bus=cxl.0,chassis=0,slot=0,port=0
-device cxl-type2,bus=hb0rp0,volatile-memdev=t2mem0,id=cxl-t2,fmw-index=0
-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191
-device cxl-rp,id=hb1rp0,bus=cxl.1,chassis=0,slot=2,port=0
-device cxl-type2,bus=hb1rp0,volatile-memdev=t2mem1,id=cxl-t2b,fmw-index=1
-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G

The guest kernel needs CONFIG_CXL_REGION_INVALIDATION_TEST=y. Without it,
the cxl_bus reset fails in an x86 guest with "failed to prepare CXL bus
reset ranges: -6".

I drove both devices with a small test module based on
tools/testing/cxl/test/accel.c. It attaches each firmware-committed
region with devm_cxl_probe_mem() and has no reset handlers. Then, with
lockdep on:

echo cxl_bus > /sys/bus/pci/devices/0000:36:00.0/reset_method
echo 1 > /sys/bus/pci/devices/0000:36:00.0/reset
echo 0 > /sys/bus/pci/slots/2/power # second device, 0000:c0:00.0

For the hang, add msleep(10000) after cxl_region_quiesce_lock() in
pci_cxl_reset_prepare(). Start the cxl_bus reset, remove 0000:c0:00.0
through sysfs two seconds later, and power off slot 0 two seconds after
that.

Separately, pci_cxl_reset_prepare() returns with the quiesce held on its
success path only, and cxl_restore_state_after_pci_reset() releases it.
With Clang 23 and CONTEXT_ANALYSIS=y passed to make, that gives five
cxl_region_quiesce warnings in drivers/pci/cxl.c.

[1]: https://lore.kernel.org/all/20261106035911.1280066-1-mhonap@xxxxxxxxxx/

DJ