Re: [PATCH v17 00/19] PCI/CXL: Add CXL reset support for Type 2 devices

From: Dave Jiang

Date: Fri Nov 20 2026 - 18:28:21 EST




On 11/20/26 12:32 AM, Srirangan Madhavan wrote:
> On 11/16/26 11:11 AM, Dave Jiang wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> On 11/5/26 11:58 AM, Srirangan Madhavan wrote:
>>> Hi folks,
>>>
>>> This series adds CXL Reset for Type 2 devices through the existing PCI
>>> reset_method and reset sysfs attributes. It disables CXL.cache, performs
>>> supported cache writeback, initiates CXL Reset with Memory Clear, and
>>> restores cached HDM decoder and CXL Device DVSEC state before PCI use
>>> resumes. Active HDM ranges are reserved and CPU caches invalidated before
>>> and after reset. The method remains limited to eligible single-function
>>> devices with both CXL.cache and CXL.mem enabled. CXL bus reset restores
>>> the same cached state and flushes the host and device caches around the
>>> reset.
>>>
>>> The reset sequence follows the CXL 4.0 specification [4], including
>>> CXL.cache disable, optional cache writeback, CXL Reset Memory Clear,
>>> ResetComplete polling, and ResetError reporting. Sections 9.7.2 and 9.7.3
>>> define the completion timeout encodings and reset access restrictions.
>>>
>>> The userspace ABI is the existing PCI reset interface:
>>>
>>>      /sys/bus/pci/devices/.../reset_method
>>>      /sys/bus/pci/devices/.../reset
>>>
>>> Userspace selects "cxl_reset" in reset_method and triggers the reset
>>> through reset. Following Dan's v6 feedback, this uses the PCI interface
>>> instead of adding a memdev-specific sysfs ABI.
>>>
>>> For sysfs reset, the generic PCI path invokes the driver's reset_prepare()
>>> callback when provided and disables PCI I/O, memory decoding, and bus
>>> mastering. The CXL path reserves affected HDM ranges exclusively and
>>> blocks IOMMU translations where applicable. It does not offline memory
>>> or revoke mappings: userspace is expected to offline CXL.mem users before
>>> reset. The reservation enforces this for claimed ranges by failing the
>>> reset while a conflicting resource is claimed [9].
>>>
>>> The global region lock is held from snapshot through restoration. Memory
>>> Clear completion is awaited for the timeout the device advertises, up to
>>> 100 seconds, so the lock can be held for that long plus cache flushes.
>>
>> I think holding it across the bus reset can deadlock with pciehp and a
>> PCI remove. I tested in QEMU with two cxl-type2 devices from Manish's
>> RFC [1].
>>
>> For a device in a pciehp slot, pci_reset_bus_function() ends up in
>> pciehp_reset_slot(), which takes the slot's reset_lock. The reset still
>> holds cxl_rwsem.region at that point.
>>
>> Two other tasks can complete a cycle. pciehp_ist() for that slot can
>> hold reset_lock and wait for pci_rescan_remove_lock in
>> pciehp_unconfigure_device(). A PCI remove of a second Type 2 device can
>> hold pci_rescan_remove_lock while cxl_decoder_detach() waits for
>> cxl_rwsem.region.
>>
>> On v17 as posted, lockdep reports the cycle:
>>
>>    [   60.891789] WARNING: possible circular locking dependency detected
>>    [   60.893014] irq/24-pciehp/103 is trying to acquire lock:
>>    [   60.893367] ffff888072b4f128 (&cxl_memdev_key){+.+.}-{4:4}, at: device_del+0x8d/0x580
>>                   but task is already holding lock:
>>    [   60.894498] ffffffff84699e80 (pci_rescan_remove_lock){+.+.}-{4:4}, at: pciehp_unconfigure_device+0x93/0x210
>>    [...]
>>                   -> #3 (&ctrl->reset_lock){++++}-{4:4}:
>>    [   60.916794]        down_write_nested+0x80/0x170
>>    [   60.917826]        pciehp_reset_slot+0x6c/0xc0
>>    [   60.918893]        pci_reset_hotplug_slot+0x5c/0x90
>>    [   60.919928]        pci_reset_bus_function+0x161/0x330
>>    [   60.920951]        cxl_reset_bus_function+0x1c6/0x280
>>    [   60.921958]        __pci_reset_function_locked+0x96/0x170
>>    [   60.922990]        pci_reset_function+0xc8/0x180
>>    [   60.923977]        reset_store+0x84/0xd0
>>    [...]
>>                   -> #2 (cxl_rwsem.region){++++}-{4:4}:
>>    [...]
>>    [   61.013490] Chain exists of:
>>                     &cxl_memdev_key --> &ctrl->reset_lock --> pci_rescan_remove_lock
>>
>> With a test-only msleep() after cxl_region_quiesce_lock() in
>> pci_cxl_reset_prepare(), it turns into a real hang. The reset, a sysfs
>> remove of the second device and a power off of the first device's slot
>> all got stuck.
>>
>> Two region reads blocked behind the reset as well. A read of
>> region1/commit timed out after 5 s, and cxl list -R stayed in D state in
>> decoders_committed_show().
>>
>> To reproduce, I used Manish's RFC as posted. It allows one cxl-type2 per
>> pxb-cxl, so each device has its own host bridge and window. On q35 with
>> cxl=on, plus the memory backends:
>>
>>    -device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=53
>>    -device cxl-rp,id=hb0rp0,bus=cxl.0,chassis=0,slot=0,port=0
>>    -device cxl-type2,bus=hb0rp0,volatile-memdev=t2mem0,id=cxl-t2,fmw-index=0
>>    -device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=191
>>    -device cxl-rp,id=hb1rp0,bus=cxl.1,chassis=0,slot=2,port=0
>>    -device cxl-type2,bus=hb1rp0,volatile-memdev=t2mem1,id=cxl-t2b,fmw-index=1
>>    -M cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=4G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=4G
>>
>> The guest kernel needs CONFIG_CXL_REGION_INVALIDATION_TEST=y. Without it,
>> the cxl_bus reset fails in an x86 guest with "failed to prepare CXL bus
>> reset ranges: -6".
>>
>> I drove both devices with a small test module based on
>> tools/testing/cxl/test/accel.c. It attaches each firmware-committed
>> region with devm_cxl_probe_mem() and has no reset handlers. Then, with
>> lockdep on:
>>
>>    echo cxl_bus > /sys/bus/pci/devices/0000:36:00.0/reset_method
>>    echo 1 > /sys/bus/pci/devices/0000:36:00.0/reset
>>    echo 0 > /sys/bus/pci/slots/2/power     # second device, 0000:c0:00.0
>>
>> For the hang, add msleep(10000) after cxl_region_quiesce_lock() in
>> pci_cxl_reset_prepare(). Start the cxl_bus reset, remove 0000:c0:00.0
>> through sysfs two seconds later, and power off slot 0 two seconds after
>> that.
>>
>> Separately, pci_cxl_reset_prepare() returns with the quiesce held on its
>> success path only, and cxl_restore_state_after_pci_reset() releases it.
>> With Clang 23 and CONTEXT_ANALYSIS=y passed to make, that gives five
>> cxl_region_quiesce warnings in drivers/pci/cxl.c.
>>
>> [1]: https://lore.kernel.org/all/20261106035911.1280066-1-mhonap@xxxxxxxxxx/
>>
>> DJ
>
> Hi Dave,
>
> Thank you for the repro. Please let me know your opinion on the following fix.
> It also covers Alison's lock scope question.
>
> Reordering the locks won't work, since pci_reset_bus_function() itself
> takes reset_lock. So I'd like to drop the region lock before the reset starts,
> and take it only for two short windows:
>
>   pci_cxl_reset_prepare()
>       lock(region):  cxl_hdm_cache_snapshot(), set reset_pending,
>                      cxl_hdm_ranges_collect() + cxl_hdm_ranges_request()
>       cxl_hdm_quiesce_flush()                      [no lock]
>   pci_reset_bus_function()                         [no region lock]
>   cxl_restore_state_after_pci_reset()
>       lock(region):  cxl_hdm_cache_copy_decoders(), pci_cxl_restore_state(),
>                      clear reset_pending
>       cxl_hdm_quiesce_flush()                      [no lock]
>
>   cxl_decoder_commit():  reset_pending -> -EBUSY   [next to sanitize_active]
>   disable_hdm():         also update the cached global_ctrl
>   cxl_reset_execute():   same two windows for the DVSEC reset
>
> With this,
>  - If a region gets torn down mid-reset, the cache is already refreshed
>    (cxl_hdm_cache_refresh_decoder()), so restore just re-reads it with
>    cxl_hdm_cache_copy_decoders(). Patch 17 had that call, and patch 18
>    dropped it when I moved the lock into prepare.
>  - A new commit mid-reset gets -EBUSY. Regions committed before the reset
>    shouldn't be able to come online, since the exclusive
>    request_mem_region() makes the kmem and pmem claims fail, as far as I
>    can tell.


I think the direction looks good.

>
> Questions
>  1. Is reset_pending in struct cxl_hdm_info, checked in
>     cxl_decoder_commit(), OK?

Yes.

>  2. Can I keep cxl_hdm_cache_copy_decoders(), now used again, instead of
>     dropping it?

Yes.

>  3. cxl_hdm_ranges_invalidate() and cxl_hdm_quiesce_prepare() assert the
>     region lock is held. To flush outside it I'd relax that and rely on
>     the HPA reservation. OK?

Sounds reasonable.

>
> If this plan sounds okay, I can implement it this way in v18 (and rerun
> your reproducer against it)
>

Sounds good!