Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices

From: Srirangan Madhavan

Date: Fri Nov 20 2026 - 02:34:50 EST


On 11/16/26 11:11 AM, Dave Jiang wrote:
External email: Use caution opening links or attachments


On 11/5/26 11:58 AM, Srirangan Madhavan wrote:
Hi folks,

This series adds CXL Reset for Type 2 devices through the existing PCI
reset_method and reset sysfs attributes. It disables CXL.cache, performs
supported cache writeback, initiates CXL Reset with Memory Clear, and
restores cached HDM decoder and CXL Device DVSEC state before PCI use
resumes. Active HDM ranges are reserved and CPU caches invalidated before
and after reset. The method remains limited to eligible single-function
devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
the same cached state and flushes the host and device caches around the
reset.

The reset sequence follows the CXL 4.0 specification [4], including
CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
define the completion timeout encodings and reset access restrictions.

The userspace ABI is the existing PCI reset interface:

/sys/bus/pci/devices/.../reset_method
/sys/bus/pci/devices/.../reset

Userspace selects "cxl_reset" in reset_method and triggers the reset
through reset. Following Dan's v6 feedback, this uses the PCI interface
instead of adding a memdev-specific sysfs ABI.

For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
callback when provided and disables PCI I/O, memory decoding, and bus
mastering. The CXL path reserves affected HDM ranges exclusively and
blocks IOMMU translations where applicable. It does not offline memory
or revoke mappings: userspace is expected to offline CXL.mem users before
reset. The reservation enforces this for claimed ranges by failing the
reset while a conflicting resource is claimed [9].

The global region lock is held from snapshot through restoration. Memory
Clear completion is awaited for the timeout the device advertises, up to
100 seconds, so the lock can be held for that long plus cache flushes.

I think holding it across the bus reset can deadlock with pciehp and a
PCI remove. I tested in QEMU with two cxl-type2 devices from Manish's
RFC [1].

For a device in a pciehp slot, pci_reset_bus_function() ends up in
pciehp_reset_slot(), which takes the slot's reset_lock. The reset still
holds cxl_rwsem.region at that point.

Two other tasks can complete a cycle. pciehp_ist() for that slot can
hold reset_lock and wait for pci_rescan_remove_lock in
pciehp_unconfigure_device(). A PCI remove of a second Type 2 device can
hold pci_rescan_remove_lock while cxl_decoder_detach() waits for
cxl_rwsem.region.

On v17 as posted, lockdep reports the cycle:

[ 60.891789] WARNING: possible circular locking dependency detected
[ 60.893014] irq/24-pciehp/103 is trying to acquire lock:
[ 60.893367] ffff888072b4f128 (&cxl_memdev_key){+.+.}-{4:4}, at: device_del+0x8d/0x580
but task is already holding lock:
[ 60.894498] ffffffff84699e80 (pci_rescan_remove_lock){+.+.}-{4:4}, at: pciehp_unconfigure_device+0x93/0x210
[...]
-> #3 (&ctrl->reset_lock){++++}-{4:4}:
[ 60.916794] down_write_nested+0x80/0x170
[ 60.917826] pciehp_reset_slot+0x6c/0xc0
[ 60.918893] pci_reset_hotplug_slot+0x5c/0x90
[ 60.919928] pci_reset_bus_function+0x161/0x330
[ 60.920951] cxl_reset_bus_function+0x1c6/0x280
[ 60.921958] __pci_reset_function_locked+0x96/0x170
[ 60.922990] pci_reset_function+0xc8/0x180
[ 60.923977] reset_store+0x84/0xd0
[...]
-> #2 (cxl_rwsem.region){++++}-{4:4}:
[...]
[ 61.013490] Chain exists of:
&cxl_memdev_key --> &ctrl->reset_lock --> pci_rescan_remove_lock

With a test-only msleep() after cxl_region_quiesce_lock() in
pci_cxl_reset_prepare(), it turns into a real hang. The reset, a sysfs
remove of the second device and a power off of the first device's slot
all got stuck.

Two region reads blocked behind the reset as well. A read of
region1/commit timed out after 5 s, and cxl list -R stayed in D state in
decoders_committed_show().

To reproduce, I used Manish's RFC as posted. It allows one cxl-type2 per
pxb-cxl, so each device has its own host bridge and window. On q35 with
cxl=on, plus the memory backends:

-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=53
-device cxl-rp,id=hb0rp0,bus=cxl.0,chassis=0,slot=0,port=0
-device cxl-type2,bus=hb0rp0,volatile-memdev=t2mem0,id=cxl-t2,fmw-index=0
-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191
-device cxl-rp,id=hb1rp0,bus=cxl.1,chassis=0,slot=2,port=0
-device cxl-type2,bus=hb1rp0,volatile-memdev=t2mem1,id=cxl-t2b,fmw-index=1
-M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G

The guest kernel needs CONFIG_CXL_REGION_INVALIDATION_TEST=y. Without it,
the cxl_bus reset fails in an x86 guest with "failed to prepare CXL bus
reset ranges: -6".

I drove both devices with a small test module based on
tools/testing/cxl/test/accel.c. It attaches each firmware-committed
region with devm_cxl_probe_mem() and has no reset handlers. Then, with
lockdep on:

echo cxl_bus > /sys/bus/pci/devices/0000:36:00.0/reset_method
echo 1 > /sys/bus/pci/devices/0000:36:00.0/reset
echo 0 > /sys/bus/pci/slots/2/power # second device, 0000:c0:00.0

For the hang, add msleep(10000) after cxl_region_quiesce_lock() in
pci_cxl_reset_prepare(). Start the cxl_bus reset, remove 0000:c0:00.0
through sysfs two seconds later, and power off slot 0 two seconds after
that.

Separately, pci_cxl_reset_prepare() returns with the quiesce held on its
success path only, and cxl_restore_state_after_pci_reset() releases it.
With Clang 23 and CONTEXT_ANALYSIS=y passed to make, that gives five
cxl_region_quiesce warnings in drivers/pci/cxl.c.

[1]: https://lore.kernel.org/all/20261106035911.1280066-1-mhonap@xxxxxxxxxx/

DJ

Hi Dave,

Thank you for the repro. Please let me know your opinion on the following fix.
It also covers Alison's lock scope question.

Reordering the locks won't work, since pci_reset_bus_function() itself
takes reset_lock. So I'd like to drop the region lock before the reset starts,
and take it only for two short windows:

pci_cxl_reset_prepare()
lock(region): cxl_hdm_cache_snapshot(), set reset_pending,
cxl_hdm_ranges_collect() + cxl_hdm_ranges_request()
cxl_hdm_quiesce_flush() [no lock]
pci_reset_bus_function() [no region lock]
cxl_restore_state_after_pci_reset()
lock(region): cxl_hdm_cache_copy_decoders(), pci_cxl_restore_state(),
clear reset_pending
cxl_hdm_quiesce_flush() [no lock]

cxl_decoder_commit(): reset_pending -> -EBUSY [next to sanitize_active]
disable_hdm(): also update the cached global_ctrl
cxl_reset_execute(): same two windows for the DVSEC reset

With this,
- If a region gets torn down mid-reset, the cache is already refreshed
(cxl_hdm_cache_refresh_decoder()), so restore just re-reads it with
cxl_hdm_cache_copy_decoders(). Patch 17 had that call, and patch 18
dropped it when I moved the lock into prepare.
- A new commit mid-reset gets -EBUSY. Regions committed before the reset
shouldn't be able to come online, since the exclusive
request_mem_region() makes the kmem and pmem claims fail, as far as I
can tell.

Questions
1. Is reset_pending in struct cxl_hdm_info, checked in
cxl_decoder_commit(), OK?
2. Can I keep cxl_hdm_cache_copy_decoders(), now used again, instead of
dropping it?
3. cxl_hdm_ranges_invalidate() and cxl_hdm_quiesce_prepare() assert the
region lock is held. To flush outside it I'd relax that and rely on
the HPA reservation. OK?

If this plan sounds okay, I can implement it this way in v18 (and rerun
your reproducer against it)

--
Regards,
Srirangan