Re: [PATCH v6 07/27] vfio/pci: Preserve vfio-pci device files across Live Update

From: Vipin Sharma

Date: Tue Nov 17 2026 - 16:59:16 EST


On Thu, Nov 05, 2026 at 07:12:17PM +0000, David Matlack wrote:
> On 2026-11-02 07:36 AM, Vipin Sharma wrote:
> > * TOCTOU synchronization: vfio_pci_dev_is_frozen() is checked locklessly
> > on entry to the ioctl, read/write and mmap paths, and re-checked under
> > a lock that freeze() holds before any operation that could change the
> > device or vconfig: under vdev->igate in vfio_pci_ioctl_set_irqs() and
> > around every config-space write in vfio_pci_config_rw_single(), under
> > vdev->memory_lock in vfio_pci_runtime_pm_entry(),
> > vfio_pci_ioctl_reset() and the SR-IOV enable path of
> > vfio_pci_core_sriov_configure(), and under dev_set->lock in
> > vfio_pci_dev_set_hot_reset(). A racing operation thus either completes
> > before freeze() takes its snapshot or fails, while existing BAR mmaps,
> > ioeventfds and dma-bufs keep working. Serializing config writes on
> > igate covers the PCI_COMMAND, INTx disable emulation, power state and
> > FLR paths at one point (the INTx disable emulation now uses the
> > igate-held __vfio_pci_intx_mask() and __vfio_pci_intx_unmask()
> > helpers), and also serializes the irq_type and virq_disabled accesses
> > of config writes with VFIO_DEVICE_SET_IRQS.
>
> I think there's a simpler way to do this. Checking the frozen flag
> locklessly and then re-checking it under whichever lock covers each
> path spreads the synchronization across igate, memory_lock and
> dev_set->lock. That makes it hard to convince yourself every path that
> can change the device is covered. It also makes every config space
> write take igate, for all vfio-pci users, even when
> CONFIG_VFIO_PCI_LIVEUPDATE=n.
>
> Instead, I suggest adding a per-device rw_semaphore to struct
> vfio_device in the VFIO core, e.g. liveupdate_rwsem, that protects a
> liveupdate_frozen flag:
>
> - vfio_device_fops_unl_ioctl(), vfio_device_fops_read() and
> vfio_device_fops_write() take liveupdate_rwsem for read and fail
> with -EIO if the device is frozen. That covers every device ioctl in
> one place, including the ones handled by the VFIO core
> (VFIO_DEVICE_FEATURE, VFIO_DEVICE_GET_REGION_INFO and
> VFIO_DEVICE_{ATTACH,DETACH}_IOMMUFD_PT). This patch doesn't block
> the last two at all today.
>
> - freeze() and unfreeze() take liveupdate_rwsem for write, before any
> other lock, and flip the flag. preserve() doesn't need it since it
> only serializes state that doesn't change after the device is
> opened.
>
> Putting this in the VFIO core also means other VFIO drivers can reuse
> it if they add Live Update support in the future.
>
> The lock order then becomes:
>
> liveupdate_rwsem -> dev_set->lock -> igate -> memory_lock
>
> freeze() can still take igate and memory_lock to check irq_type,
> pm_runtime_engaged and sriov_active. But the ioctl, read and write
> paths no longer need to re-check anything under those locks. So you
> can drop the igate change in vfio_pci_config_rw_single(), the
> open-coded vfio_pci_zap_and_down_write_memory_lock() in
> vfio_pci_runtime_pm_entry() and vfio_pci_ioctl_reset(), and the
> smp_load_acquire()/smp_store_release() pair.
>
> Two paths still need their own handling because they don't go through
> the device's file:
>
> - vfio_pci_core_sriov_configure() is reached through sysfs. Take
> liveupdate_rwsem for read around the frozen check and the
> sriov_active update.
>
> - VFIO_DEVICE_PCI_HOT_RESET issued on one device also resets the other
> devices in its dev_set. Keep the check in
> vfio_pci_dev_set_hot_reset() under dev_set->lock. freeze() sets the
> flag while holding both liveupdate_rwsem and dev_set->lock, so
> holding either one is enough to read it. Please don't take the other
> devices' liveupdate_rwsem there, since that would invert the lock
> order relative to freeze().
>
> I don't think mmap needs to be blocked at all. Creating a new mapping
> doesn't change the device or any of the state that gets preserved, and
> existing mappings already keep working while the device is frozen.
> Leaving mmap alone also avoids taking liveupdate_rwsem under mmap_lock,
> which could deadlock: the ioctl, read and write paths can fault on user
> memory (taking mmap_lock) while holding liveupdate_rwsem for read, and
> freeze() may be waiting for the write lock. Please update the comment
> in freeze() and the DOC section to match.
>

Thanks for suggestion, this is indeed simpler.

> > diff --git a/drivers/vfio/pci/vfio_pci_config.c b/drivers/vfio/pci/vfio_pci_config.c
> > index 9914f3ac69ae..1f1ac7a798aa 100644
> > --- a/drivers/vfio/pci/vfio_pci_config.c
> > +++ b/drivers/vfio/pci/vfio_pci_config.c
> > @@ -644,10 +644,10 @@ static int vfio_basic_config_write(struct vfio_pci_core_device *vdev, int pos,
> >
> > if (virt_intx_disable && !vdev->virq_disabled) {
> > vdev->virq_disabled = true;
> > - vfio_pci_intx_mask(vdev);
> > + __vfio_pci_intx_mask(vdev);
> > } else if (!virt_intx_disable && vdev->virq_disabled) {
> > vdev->virq_disabled = false;
> > - vfio_pci_intx_unmask(vdev);
> > + __vfio_pci_intx_unmask(vdev);
> > }
> > }
>
> With the rwsem, config writes no longer need igate, so this change
> isn't needed. If config writes and VFIO_DEVICE_SET_IRQS really do race
> on irq_type and virq_disabled today, that's unrelated to Live Update.
> Please send it as a separate fix.
>

This will go away.

> Lastly, please add the freeze infrastructure and the checks in their
> own commit ahead of this one, e.g. "vfio: Add infrastructure to freeze
> devices for Live Update". Then this patch only has to implement
> freeze() and unfreeze() on top of it, which will make both easier to
> review.

Sounds good. Thanks for the review.